Starting a VM from a MacOS sandbox | Brent Fitzgerald

7 min read Original article ↗

This is a quick tech note about using Lima to start VMs from MacOS sandboxed processes.

VMs with Lima

Increasingly often I have multiple worktrees in progress, each with an agent working on a specific feature. To isolate all service dependencies and ports, I’ve been starting to use throwaway Linux VMs with Lima, and then I’ll run docker compose and other app stuff inside of the VM.

I decided it would be really helpful to make my coding agents also do this. That way they can do a full development and test lifecycle in full isolation without any docker compose port collisions or shared databases.

Sandboxes with Seatbelt and nono

I run each coding agent in a sandbox. I am a little suspicious of the agent-provided sandboxes, which may change over time and seem a bit too clever with options like auto mode. So I use a separate system.

On a Mac, sandboxing typically uses Apple’s Seatbelt, the kernel-level engine used by sandbox-exec. A seatbelt profile is a bunch of very explicit rules about what to deny and allow.

My sandbox uses a Seatbelt profile that I initially constructed with Agent Safehouse.

However, I’ve also been recently experimenting with nono, a capability-based sandbox that also uses Seatbelt under the hood. Theoretically nono can make it easier to compose capabilities (via groups) and to inherit from and override readymade profiles.

Lima and the Virtualization entitlement

Lima’s driver on Apple Silicon is vz, which uses Apple’s Virtualization framework. The framework requires an entitlement. limactl already ships with it. Here’s how you can see it:

$ codesign -d --entitlements - /opt/homebrew/bin/limactl
...
[Key] com.apple.security.virtualization
[Value]
    [Bool] true

So the binary is allowed to make VMs. So far so good!

The failure

Unfortunately running limactl start with my sandbox profile failed with an error:

[hostagent] Starting VZ
[hostagent] Setting up Rosetta share
fatal: Error Domain=VZErrorDomain Code=1 Description="Internal Virtualization error. The virtual machine failed to start."

Plus some other issues:

failed to detect system DNS, falling back to [8.8.8.8 1.1.1.1]  error="open /etc/resolv.conf: operation not permitted"

The DNS warning is just an issue of read access on /etc/resolv.conf, and there’s nothing sensitive in there. The real problem is VZErrorDomain Code=1. It says “Internal Virtualization error” but does not say what was denied exactly.

Since the error occurred right after “Setting up Rosetta share” I thought at first it looked like a Rosetta problem (which turned out to not be the case).

The Seatbelt denial was not in Lima’s logs. Lima only sees the generic VZ error. To find the Seatbelt logs I had to look at the MacOS unified log, written by the kernel.

My initial log show queries found nothing. The denials are emitted by the kernel in a specific format and it was hard to figure out the right predicate to use. I eventually found it by filtering on the literal Sandbox: prefix:

log stream --style compact --predicate 'eventMessage CONTAINS "Sandbox: "'

I tested this by triggering a denial I understood: read a blocked file under the sandbox. The log showed it:

Sandbox: wc(89150) deny(1) file-read-data /private/etc/sudoers

Good. Logging works. The format is Sandbox: <process>(<pid>) deny(1) <operation> <detail>.

The denials

With the stream running, I started the VM again. Here’s the deny I saw:

Sandbox: limactl(42303) deny(1) mach-lookup com.apple.Virtualization.VirtualMachine (xpc)

So to start a VM, the Virtualization framework tries to look up an XPC service named com.apple.Virtualization.VirtualMachine. That service is the process that actually hosts the guest. The sandbox did not allow the lookup, the lookup failed, and the framework returned Code=1 with no detail.

After some research and with agent assistance, I learned there’s a way to write a rule to allow a specific mach-lookup:

(allow mach-lookup
    (global-name "com.apple.Virtualization.VirtualMachine"))

Then I tried again. I got past that denial, and more denials appeared, in sequence, as we got further:

limactl deny(1) mach-task-name others [com.apple.Virtualization.Virtual(44215)]
limactl deny(1) generic-issue-extension extension-class:com.apple.virtualization.extension.fuse
limactl deny(1) generic-issue-extension extension-class:com.apple.virtualization.extension.rosetta-directory-share

Each one is a step in starting the VM:

  • mach-task-name took me a bit. It is the framework getting a handle to the host process it just spawned. That process runs outside the sandbox since launchd starts it. The parent needs its task-name to monitor it.
  • The fuse extension is the virtio-fs directory share, used to mount the host filesystem into the VM.
  • The rosetta-directory-share extension is the Rosetta mount that lets the guest run x86-64 binaries. The VM boots without it, but Rosetta will not work in the guest unless you allow it.

Seatbelt rules to enable Lima VM start

Here are the four things I had to change, in Seatbelt form. Three are virtualization-specific:

(allow mach-lookup
    (global-name "com.apple.Virtualization.VirtualMachine"))

(allow mach-task-name)

(allow generic-issue-extension
    (extension-class "com.apple.virtualization.extension.fuse")
    (extension-class "com.apple.virtualization.extension.rosetta-directory-share"))

The fourth was just file access. Lima keeps per-instance state and sockets under ~/.lima, and downloads guest images into ~/Library/Caches/lima:

(allow file-read* file-write*
    (subpath "/Users/you/.lima")
    (subpath "/Users/you/Library/Caches/lima"))

The DNS warning was worth fixing too. /etc/resolv.conf is a symlink to /var/run/resolv.conf. My profile allowed the symlink path but not the real target, so the read failed and Lima fell back to other DNS. The fix is allowing a file-read:

(allow file-read*
    (literal "/private/var/run/resolv.conf"))

So I added the virtualization rules in a small block in my .sb file next to Lima file grants.

Now the VM boots to READY inside the sandbox. Yay!

One note on (allow mach-task-name) is that it’s broad, and I would rather it weren’t. Seatbelt lets you scope task-port rules to same-sandbox, but the VM host process is not in the sandbox, so that scope does not match it. I don’t see a clean way to scope to a single executable. I think the risk is small. The task name is used for introspection, but the process still cannot read the target’s memory or control its execution.

In nono

nono’s model is higher level so you don’t have to touch the s-expression syntax. It’s files, network, and credentials. You grant directories and domains, but I couldn’t find schema support for mach-lookup or a sandbox extension, and no built-in group for Lima or virtualization.

However, since it uses Seatbelt under the hood on Macs, nono does expose unsafe_macos_seatbelt_rules to provide custom, untyped rules. It’s a list of raw Seatbelt S-expressions applied verbatim. The file grants map to normal nono capabilities. The three virtualization rules go in there:

{
  "filesystem": {
    "allow": ["~/.lima", "~/Library/Caches/lima"],
    "read_file": ["/private/var/run/resolv.conf"]
  },
  "unsafe_macos_seatbelt_rules": [
    "(allow mach-lookup (global-name \"com.apple.Virtualization.VirtualMachine\"))",
    "(allow mach-task-name)",
    "(allow generic-issue-extension (extension-class \"com.apple.virtualization.extension.fuse\") (extension-class \"com.apple.virtualization.extension.rosetta-directory-share\"))"
  ]
}

So the Lima file grants are straightforward, but letting the child process use an XPC host service and get task names is the part where we need raw Seatbelt.

I’ve looked into contributing this as a nono “pack”, but currently nono doesn’t support composing multiple profiles, so someone using this couldn’t say “I want claude + lima support.” nono’s composition mechanism seems to be groups, but because groups are designed to be either included or excluded, they don’t support the generic unsafe_macos_seatbelt_rules entry available to profiles.

Takeaway notes for other travelers

  • VZErrorDomain Code=1 doesn’t say much, but if the Virtualization framework fails to start inside a sandbox, it’s probably the denied mach-lookup.
  • Seatbelt denials are in the unified log, written by the kernel, under the Sandbox: prefix. So you can log stream looking for that.
  • The minimal set for Lima vz to start a VM via limactl: the com.apple.Virtualization.VirtualMachine mach-lookup, mach-task-name for the host process, and the two virtualization issue-extensions. Plus file access to ~/.lima and its cache.

I’ve started a nono discussion about documenting these rules, composing them into a pack, and maybe promoting them to a first-class capability.