GitHub - Katakate/k7d: ⚡ The Rust VMM that unlocked forking live Kubernetes clusters in ~100 ms ⭐ Star it if you like it!

Fork a live Kubernetes cluster.
Two clocks: ~105 ms (minimal CI guest) and ~1.1 s (Ubuntu k3s+Cilium+Tetragon).

flowchart TB
    subgraph SRC["source cluster — live, keeps running"]
        direction LR
        S0["vm-0<br/>k3s server"] --- S1["vm-1<br/>agent"] --- S2["vm-2<br/>agent"]
    end
    SRC ==> A
    SRC == "tree_fork_batch(3)<br/>~105 ms CI guest · ~1.1 s Ubuntu k3s" ==> B
    SRC ==> C
    subgraph A["fork A — own bridge, same IPs / MACs / TLS"]
        direction LR
        A0["vm-0"] --- A1["vm-1"] --- A2["vm-2"]
    end
    subgraph B["fork B"]
        direction LR
        B0["vm-0"] --- B1["vm-1"] --- B2["vm-2"]
    end
    subgraph C["fork C"]
        direction LR
        C0["vm-0"] --- C1["vm-1"] --- C2["vm-2"]
    end
    RAM[("guest RAM — one mapping<br/>parent pages shared until written")]
    A -. "dirty pages only" .-> RAM
    B -. "dirty pages only" .-> RAM
    C -. "dirty pages only" .-> RAM
    classDef src fill:#1f2d3d,stroke:#f2a33c,color:#f5f7fa,stroke-width:2px;
    classDef fork fill:#14261c,stroke:#3ddc84,color:#f5f7fa;
    classDef ram fill:#2a2438,stroke:#a78bfa,color:#f5f7fa;
    class SRC,S0,S1,S2 src;
    class A,B,C,A0,A1,A2,B0,B1,B2,C0,C1,C2 fork;
    class RAM ram;

_{Three copies of a running 3-node cluster. Nothing inside rebooted; the copies pay for the pages they change, not for the base.}

_{Recorded in real time. Guests run at ×8 (dilation: 8); fork latency chips are wall-clock. Only waiting compresses — see Time warp.}

RL training and agent evals whose environments are Kubernetes - clusters, charts, in-cluster workloads - need thousands of isolated, resettable worlds. Not one sandbox, and not a cold kind cluster per trial. Booting a fresh Kubernetes cluster takes ~30 s and full RAM per copy. k7d boots it once, then forks the live cluster. Two clocks, always together: ~105 ms for the minimal CI guest (test-enforced 1 s budget) and ~1.1 s for Ubuntu k3s+Cilium+Tetragon (48 s source bring-up). Forks share memory until they diverge, so 50 CI-guest copies cost dirty pages, not 50 × full guest RAM.

The same engine is also a great fork-first VMM when your unit is a single VM sandbox (including docker-in-VM): blazing-fast warm forks with faithful snapshotting of memory, disk, processes, and networking. For running that at scale - Kubernetes orchestrating your sandboxes, plus a CLI / API / Python SDK for agents - see the sibling project Katakate k7.

100% open‑source (Apache‑2.0). For technical support, write us at: hi@katakate.org

Why k7d

• ⚡ Fork a running Kubernetes cluster — two clocks: ~105 ms (minimal CI guest, 1 s test-enforced budget) and ~1.1 s (Ubuntu k3s+Cilium+Tetragon, 48 s source bring-up). Pack 50 CI-guest copies on one 64 GB box. Cold-booting a fresh cluster takes ~30 s and a full RAM bill per copy; k7d boots once, then copies share memory until they diverge.
• 🔁 The cluster keeps running after the fork - no agent restarts, no broken TLS, no "please wait while Kubernetes comes back." Each copy looks identical to the original from the inside.
• 🌲 Built for AI agents that explore many worlds - fork a branch, try something, keep the winners, throw away the losers. The agent decides what to keep; k7d enforces RAM and disk budgets so the tree doesn't eat the machine.
• ⏩ k7d owns the guest clock - an episode that is mostly waiting (probe periods, backoffs, reconcile ticks) does not have to wait in wall time. Lockstep clock jumps compress an 8.7-minute guest-time episode 1.83× with zero fidelity delta; the carried KVM dilation patch runs the guests at 8× wall (522 s → 79 s). Only waiting is compressible; compute stays real. Details below.
• 🧊 Also a blazing-fast VM-sandbox VMM - warm-fork a single guest in ~5 ms with faithful memory / disk / process / network state. Ideal for docker-in-VM and any workload that needs resettable isolated machines, not only whole k8s clusters. Pair with Katakate/k7 when you want k8s orchestration + Python SDKs on top.
• 🔬 We use formal methods where they pay off - Kani on selected unsafe / arithmetic paths, and Aeneas→Lean on the tree budget/eviction model. Not a claim that everything is proven - details below.
• 🪶 ≤30k lines of Rust for the VMM + shim - small enough to read and audit. Deliberately not a kitchen-sink VMM.
• ✅ Every number here is a CI assertion - not a one-off benchmark paste. If a latency claim drifts, a test fails. Methodology: the benchmark write-up.

Quickstart

You need a Linux amd64 / x86_64 host with KVM (/dev/kvm present) - same ISA (amd64 is the Debian name; tarballs use x86_64). No arm64 build yet. Five minutes, from a clone or a clean box:

git clone https://github.com/Katakate/k7d && cd k7d
sudo ./scripts/quickstart.sh

That doctor-checks /dev/kvm, /dev/vhost-vsock, /dev/net/tun, and cgroup v2; fetches the release tarball + guest artifacts if they are not already here; starts k7d; boots a busybox 3-node tree; forks 4 branches; and prints wall-clocks next to both rows of the two-clocks table. k7d doctor and k7d quickstart are the same verbs on an already-installed binary.

k7d doctor                 # host checks; does not start a daemon
k7d quickstart             # 3-node busybox + 4 forks (daemon must be up)

To build from source instead: Rust + Docker, then make release (k7d-v*-x86_64-linux.tar.gz + install.sh). The older examples/cluster-tree-search demo is still there (python3 run_demo.py --mode busybox --branches 4). To put real Kubernetes pods inside those VMs (runtimeClassName: k7), see HACKING.md. Full docs (API reference, installers) will ship separately - this README is the product pitch + getting started.

A frontier agent with a fork button

_{Second wave: one copy is about to get wrecked on purpose. 21 / 24 VMs. (whole run at 7× · the same incident seen from the tree · a depth-3 tree, scripted)}

Someone planted a miner in a Deployment that looks like a helper. The real app (k7-victim) is fine. Kimi K3 does not touch production first - it clones the cluster three ways and tries delete, scale-to-zero, and “just watch.” The miner comes back every time. So it forks again from the copy that already knows delete is a lie. One of those labs restarts k3s to catch the respawner and is left broken on purpose. No rollback. Another hangs an admission lock on the name node-agent; recreate comes back denied. Only that pair of moves is replayed on the source. Soak: still gone. Victim 2/2. Tetragon clean. The story · Video · JSONL. Dashboard: examples/k7view/. Reproduce on a KVM box: sudo make helm-pin (Cilium/Tetragon helm v3.16.4), then examples/k7d-mcp/ and gym/. First Ubuntu image: guest/ubuntu/build-node-image.sh.

Using k7d for GRPO / agent tree search

If you already have Kubernetes tasks or scenarios (a Helm chart, a set of YAML manifests, an eval harness that talks to a kube-apiserver), the shape is:

1. Boot the scenario once - bring up your cluster (or adopt a running one that k7d already hosts) and wait until it is in the state you want every rollout to start from.
2. Root a tree at that checkpoint.
3. For each GRPO group (or tree-search step): fork_batch(N) → run your N policies against the N copies → score → protect the winners, prune the losers → let the daemon auto_evict under your RAM/disk budget.
4. Roll forward from a protected winner when you want the next generation to start from a better state, or rollback to an earlier node when you don't.

flowchart LR
    boot["boot the scenario once<br/>48 s Ubuntu k3s+Cilium+Tetragon"] --> root["root checkpoint"]
    root --> fork["tree_fork_batch(N)<br/>~1.1 s per fork"]
    fork --> roll["N rollouts<br/>byte-identical starts"]
    roll --> score["your reward model"]
    score -- winners --> keep["tree_protect"]
    score -- losers --> drop["tree_prune"]
    keep --> next["next generation<br/>fork from the winner<br/>or tree_rollback"]
    drop --> evict["tree_auto_evict<br/>RAM / disk budget"]
    next --> fork
    evict -.-> fork
    classDef k7d fill:#14261c,stroke:#3ddc84,color:#f5f7fa;
    classDef you fill:#1f2d3d,stroke:#f2a33c,color:#f5f7fa;
    class fork,keep,drop,evict,next,root k7d;
    class boot,roll,score you;

_{Orange: yours. Green: k7d verbs. The trainer owns rewards and policy; the daemon owns environments and budgets.}

Why byte-identical starts matter for GRPO

GRPO (and most group-relative methods) compare rewards within a group. If member A starts from a colder cache, a different etcd revision, or a half-ready Deployment than member B, the reward gap is noise, not signal. A k7d fork is a copy of the live machine - same memory, same disk, same in-cluster TLS sessions, same kube-apiserver state. Every member of the group begins from a byte-identical world, then diverges only because of what your policy did.

That is the difference between "we reset the env" and "we cloned the universe."

How your agent talks to the tree

Your training loop owns rewards and policy. k7d owns environments and budgets. The agent talks JSON-lines over a Unix socket (/run/k7d/k7d.sock). The verbs you actually need:

A thin Python client that covers exactly this loop lives in examples/cluster-tree-search/. Treat it as the template for wiring your GRPO trainer - not as a finished SDK. To watch a live tree (git-graph, fork latency, protect/evict notes, per-branch traces) run examples/k7view/ on the node - k7d stays machine-first; k7d-view is the observer. The full API reference will live in the docs site.

How it works (the non-obvious bits)

You do not need to be a VMM engineer to use k7d. You do need to know why a sub-second cluster fork is even possible, because that is the product. Two clocks: ~105 ms on the minimal CI guest, ~1.1 s on Ubuntu k3s+Cilium+Tetragon.

flowchart TB
    TRAINER["your trainer / agent<br/>Python client · MCP server · gym"]
    SHIM["containerd-shim-k7-v1<br/>runtimeClassName: k7"]
    VIEW["k7view<br/>read-only dashboard (SSE)"]
    TRAINER -- "JSON lines over<br/>/run/k7d/k7d.sock" --> K7D
    SHIM -- "same socket" --> K7D
    K7D -. "tree_list / tree_nodes / tree_watch" .-> VIEW
    subgraph K7D["k7d daemon — one process, one address space, many VMs"]
        direction LR
        TREE["snapshot tree<br/>fork · fork_batch · rollback<br/>protect · prune · auto_evict<br/>RAM + disk budget"]
        VMM["KVM VMM<br/>vCPUs · virtio-blk / net / vsock<br/>virtiofs · dirty bitmap<br/>(CPU log ∪ device writes)"]
        CLOCK["clock owner<br/>kvmclock + per-vCPU TSC offsets<br/>lockstep warp · dilation ×N"]
        TREE --> VMM
        CLOCK --> VMM
    end
    VMM --> MEM[("guest RAM<br/>MAP_PRIVATE CoW views<br/>children share the parent's pages")]
    VMM --> BRS["bridge per cluster<br/>source"]
    VMM --> BRF["bridge per fork<br/>same guest IPs / MACs replayed"]
    VMM --> AGENT["k7-agent in every guest<br/>exec · stdio · probes over vsock"]
    classDef ext fill:#1f2d3d,stroke:#f2a33c,color:#f5f7fa;
    classDef core fill:#14261c,stroke:#3ddc84,color:#f5f7fa;
    classDef res fill:#2a2438,stroke:#a78bfa,color:#f5f7fa;
    class TRAINER,SHIM,VIEW ext;
    class K7D,TREE,VMM,CLOCK core;
    class MEM,BRS,BRF,AGENT res;

A fork, end to end, is a short critical section on the source:

sequenceDiagram
    participant T as trainer
    participant D as k7d
    participant S as source VMs
    participant C as child VMs
    T->>D: tree_fork_batch(n)
    D->>S: pause every vCPU, drain in-flight virtio I/O
    D->>D: read dirty bitmap = KVM dirty log ∪ device-written pages
    D->>C: map parent RAM copy-on-write, copy only the dirty pages
    D->>C: new bridge per child, replay guest IPs / MACs
    D->>C: restore vCPUs, re-arm timers, KVM_SET_CLOCK (no time jump backwards)
    D->>S: resume
    D-->>T: n children + latency_ns
    Note over S,C: from inside, nothing moved:<br/>same addresses, same TLS, same TCP

Memory is shared until someone writes. Guest RAM lives in one file. A fork pauses the source for a moment, notes which pages changed since the last checkpoint, maps the child's memory as a copy-on-write view of the parent's, and copies only those dirty pages. Everything else is shared. That is why 50 forks of a cluster fit in 64 GB: you pay for divergence, not for the base.

The cluster does not reboot because the network lies consistently. Each cluster lives on its own private Linux bridge. A fork gets a new bridge with the same guest IPs and MAC addresses as the source. From inside the guest, nothing moved - same addresses, same ARP cache, same TLS certs, same established TCP - so kubelet, the CNI, and the control plane keep running. Separate bridges mean forks cannot see each other. Without this, every fork would force a kubelet restart and a ~1-2 s agent restart per node, and the ~105 ms / ~1.1 s claims would be impossible.

Forks live in a tree the daemon manages under budget:

flowchart LR
    R["base cluster<br/>root"] --> A["fork A"]
    R --> B["fork B<br/>pruned: low reward"]
    R --> C["fork C"]
    A --> A1["fork A1<br/>protected: winner"]
    C --> C1["fork C1<br/>evicted under RAM budget"]
    C -- "tree_rollback" --> C2["fork C'<br/>retry from C"]
    classDef live fill:#14261c,stroke:#3ddc84,color:#f5f7fa;
    classDef win fill:#14261c,stroke:#f2a33c,color:#f5f7fa,stroke-width:2px;
    classDef dead fill:#1a1a1a,stroke:#555,color:#999,stroke-dasharray: 4 3;
    class R,A,C,C2 live;
    class A1 win;
    class B,C1 dead;

Any live node is a fork source, so trees go as deep as the search does (depth 3 in the scripted clip; the daemon reports real parent edges, the UI does not invent them).

Three tricks that had to be right

These are the kinds of bugs that silently break "byte-identical." CHALLENGES.md has nearly 200 of them; these three are the headline ones:

1. Device writes are invisible to the hypervisor's dirty log. Block devices in k7d write guest memory from userspace. The hypervisor only sees CPU writes - so a fork taken after disk I/O would resurrect pre-I/O bytes on those pages (silent corruption). Fix: track device dirty pages ourselves and merge them into the fork bitmap, and drain in-flight I/O before the bitmap is read. (CHALLENGES.md #43)
2. A restored guest with a blank timer chip freezes time. After fork, the interval timer was left unprogrammed - no timer interrupts, CLOCK_REALTIME stuck, and Kubernetes quietly parks. Fix: re-arm the timer (and reset the paravirtual clock) on every restore. (CHALLENGES.md #40)
3. Identical IPs only work if the L2 domains are separate. Replay the source's addresses onto a fresh bridge per fork. Same view from inside; no conflicts across forks. That is the zero-restart trick above.

Time warp: only waiting is compressible

Fork removes the cost of starting an episode. Most of what is left in a Kubernetes episode is waiting - readiness probes, CrashLoopBackOff timers, controller reconcile ticks. A 3-node Ubuntu cluster is all-idle ~70% of the time and only 7-10% of its vCPUs are awake. k7d already owns the guest clock end to end (kvm-clock, no RTC, no NTP client, KVM_SET_CLOCK on every restore), so the same pause that makes a fork correct can also move every member's clock by the same delta:

The gym exposes this as autowarp_mode per lease: off / jump / micro / dilate. Forks of a dilated cluster land on the source's clock and can be born at a different factor (dilation on tree_fork_batch), so one ×1 source tree serves every mode. Forward only, lockstep only, whole-cluster pause or nothing - a declined warp is always correct. Reproduce with make remote-warp-bench and make remote-dilation-install.

Measured numbers

On one bare-metal box (~€40/month bare-metal: Ryzen 5 3600, 6 cores, 64 GiB, NVMe):

Every row is an integration-test assertion (LATENCY_BUDGETS.md). Full methodology: the benchmark write-up.

Kubernetes feature support

Two layers - most RL users only care about the first.

Inside a forked cluster (your GRPO scenario)

This is the k3s that lives inside the VMs you fork. The fork engine is N-node (tree_create_cluster(vm_count) / adopt any live set) - there is no hard-coded 3. The CI fixture that proves the headline numbers is a 3-node control plane with flannel + kube-proxy and a real in-cluster Deployment. That default profile keeps stock k3s add-ons off so the lean path stays lean; each add-on below is an opt-in profile on the same switchboard, with its own proving test. Status below mixes “API can do it” with “fixture exercises it.”

Need CoreDNS + Ingress on one cluster? Flip the profile flags - the switchboard is configuration, not a redesign. Same for a larger CI fixture: wiring 20 nodes is configuration + RAM, not a new fork feature.

Host RuntimeClass (pods as k7d VMs)

This is the outer layer: kubectl on the host schedules pods into k7d microVMs via runtimeClassName: k7. Relevant if you also want single-VM sandboxes, not only whole-cluster forks.

Why not Firecracker / Kata / E2B-style sandboxes?

They fork a sandbox. k7d forks a VM or an entire cluster. Deliberately not E2B-API compatible - different job. forkd is the honest nearest neighbor on the sandbox side: same CoW idea, one Firecracker process per child, no cluster unit. CubeSandbox is the honest nearest neighbor on density and packaging. Neither is a cluster time machine.

Security model & limitations

• One daemon, many VMs, one address space. Live copy-on-write fork requires parent and child memory to be mappings in the same process. Guest→host isolation is still KVM; isolation between sibling forks of the same tenant is weaker than Firecracker's one-jailed-process-per-VM. k7d is built for fleets of your own environments, not hostile multi-tenant isolation between forks.
• What survives a fork: everything inside the forked set - in-cluster TLS, established TCP between member VMs, disk state. Guest clocks are reset so time does not jump backwards.
• What doesn't: TCP to the outside world (the far end never forked). Forks keep their in-cluster addresses; only the host-facing identity is new.
• Single host, x86_64 Linux + KVM only. Cross-node fork is on the roadmap. Does not build on macOS/Windows.
• Young project. ≤30k LOC for the VMM + shim, one primary test machine, no security audit yet. Small surface by design.

Formal verification

Selected critical pieces are machine-checked - not the whole runtime:

make kani                      # selected unsafe / arithmetic harnesses
make verif-gen verif-build     # regenerate Lean model + prove it

Roadmap

• ✅ Warm single-VM fork, snapshot trees, budget eviction
• ✅ containerd shim + runtimeClassName: k7
• ✅ Live 3-node cluster fork — two clocks: ~105 ms CI guest / ~1.1 s Ubuntu k3s+Cilium+Tetragon
• ✅ Kani + Aeneas proofs on selected critical paths
• ✅ Guest time warp: lockstep jumps (1.83×, zero fidelity delta) and ×8 KVM dilation, fork-aware
• ✅ Prebuilt release tarball + install.sh (make release)
• 🔜 Config file (/etc/k7d/config.toml)
• 🔮 Cross-node fork

License

Apache-2.0 - see LICENSE.