Ask HN: What's the most secure way to run a local model?
Given the recent “attention” in the space - what are your preferred tools and techniques for running models such as K2.7 in a secure manner.
1 thread
- litellm listens on 127.0.0.1:4242 with some pre-shared key, give the credentials to the harness. - Tools can exfiltrate any data accessible to them given network access, so all tools run inside a VM or a container targeted to the specific tasks. For example, I've made podman spawner that runs a 'tool server' inside a container, and all of my custom harness' tool requests go to that container (which can be barred from the network). The workspace for that LLM is solely inside the ephemeral container, but it can get bind mounts from outside if needed.