Where Rust Ends and the Kernel Begins

Choosing Rust for a security-critical microkernel feels obvious on paper. Replace C with Rust and whole classes of memory bugs disappear. That story is true, and it shaped every engineering decision we made at Foundation building KeyOS, the operating system for our Passport Prime device. At its core is Palladium, a Rust microkernel we derived from the Xous microkernel originally built by Bunnie Huang and Sean Cross, and evolved substantially for our security and embedded requirements. Building Palladium taught us that memory safety in an operating system is a system property, not a language property alone. The compiler can only see one program at a time.

What Rust Buys Inside a Process

Inside a single process, Rust’s value in kernel code is real and measurable. References have lifetimes, mutable aliases are statically rejected, and resources clean up deterministically via Drop. In OS code, where many failures are cleanup failures rather than logic errors, this matters more than it does in application software. Rust also changed how we write interfaces. Palladium’s IPC ABI is ultimately built from message IDs, scalar arguments, and memory ranges, but the application-facing layer doesn’t have to look like raw registers. Typed wrappers turn message opcodes into enums; a Buffer abstraction can own page-aligned IPC memory and unmap it when dropped. These aren’t only ergonomic wins—they reduce the amount of code that has to manually remember protocol invariants. Rust also makes security review more tractable: unsafe code doesn’t disappear from a kernel, but it becomes locatable. A page-table manipulation or a raw slice over an IPC buffer stands out visually from ordinary logic, so reviewers can spend attention where the compiler’s proof has stopped.

Where the Compiler’s View Ends

The hard boundary is IPC. Inside one process, Rust can prove that &mut T is exclusive and that &T doesn’t outlive what it references. Across processes, those types don’t cross the boundary. What crosses is a message: a server ID, scalar values, and sometimes a page-aligned memory range. Once a buffer is sent to another address space, there’s no lexical lifetime for the compiler to check. The receiver is separately compiled, may not be Rust, may crash or die while holding memory, and its actions are outside the compiler’s visibility entirely. The operating-system question is not “can Rust express &mut T?” It can. The question is: what is the cross-process runtime equivalent of &mut T, and who enforces it against code the compiler cannot see?

The MMU as Borrow Checker

Palladium’s answer is structural: ownership becomes page-table state. When one server lends a buffer to another, we don’t pass a pointer—we rewrite page tables. At send time, the kernel revokes the sender’s mapping: the leaf page-table entry becomes unmapped-but-reserved. The kernel still tracks the underlying physical page so the lease can be validated and returned later, but the sender can no longer read or write through it. The kernel then installs the page at a freshly chosen virtual address in the receiver’s space, mapped read-write, and records the provenance. The MMU enforces what the borrow checker would enforce inside a single program—it just does it between programs, in hardware, at the page-table level.

The ordering matters. The sender’s access is revoked before the receiver’s is granted—a break-then-make discipline designed to prevent the same writable page from appearing in two address spaces at once. On return, the kernel checks that the physical page coming back is the same one that was lent. If a buggy or malicious peer tries to return a different page, the kernel refuses with a sharing violation. A peer cannot substitute memory into the client’s address space by manufacturing a plausible-looking message. This is the clearest example of the central point: Rust gave Palladium the language of ownership. The kernel had to make that ownership real at the hardware level, in the places Rust cannot reason about.

Four Ownership Shapes in IPC

We expose four memory-bearing message kinds that map directly to Rust’s ownership shapes. Move transfers ownership permanently—the sender’s page is emptied, the receiver gets it outright, and touching the old address faults, the runtime parallel to a moved-from value being a compile error. Borrow and MutableBorrow are temporary leases that block the sender until the same physical page returns, validated by the kernel. BlockingMove is the most subtle: semantically it’s a synchronous request-and-response by value, but the kernel implements the outbound leg as a lend. The reason is that the sending process may have other threads. If we fully freed the sender’s virtual address range while the caller blocked, another thread could allocate that range for unrelated memory, and restoring the range on failure would corrupt live state. By lending rather than moving, we keep the sender’s range reserved-but-inaccessible until the operation completes. That’s not a problem the borrow checker was designed to solve. It spans processes, threads, virtual address allocation, and error recovery simultaneously.

Typed Calls Still Need Runtime Validation

Higher-level API crates make all of this look like typed method calls. A caller serializes a value into a page-aligned Buffer, then chooses send, lend, lend_mut, or blocking_move. Those names preserve the ownership decision at the call site and map directly to the kernel’s verbs. We use rkyv for zero-copy access to typed data in those pages: the page remap avoids copying the payload through the kernel, while rkyv lets the receiver read the archived structure in place. But zero-copy doesn’t mean zero checking. The receiver is looking at bytes that came from another protection domain. Before exposing a typed view, the library validates the archive. That’s another instance of the same boundary: Rust can make the typed view safe once it exists; the system has to decide whether the untrusted bytes are valid enough to become that view in the first place.

Page Granularity as a Design Tradeoff

The unit of MMU enforcement is a page—not a Rust value, a struct, or a byte slice. That makes the mechanism simple and hardware-enforced, but coarser than the logical payload. Lending a 64-byte request lends the surrounding 4 KiB page. The offset and valid-length fields are advisory metadata; a correct receiver honors them, but the MMU doesn’t. Once a page is mapped into a peer, that peer can inspect the whole page. This is a confidentiality problem, not just a performance detail. In-process, a &[u8] of length n exposes exactly n bytes to safe Rust code. Across processes, the page-table boundary enforces who owns the page and whether the receiver can write it, but it doesn’t enforce sub-page length. You can get finer granularity by copying exactly n bytes into a fresh zeroed buffer, but that’s a copy. Capability hardware or memory tagging could shift this boundary on different architectures. On our current hardware—a single-core Cortex-A—page granularity is the hard protection unit.

Failure Is a Memory-Safety Problem

One of the most important lessons from building Palladium is that failure handling is part of memory safety. In ordinary Rust, a borrow has a lexical lifetime. In a microkernel, a borrow has a queue entry, a waiting sender, a receiver-side virtual address, a sender-side reservation, and a cleanup rule. If the server queue is full, the kernel must detect that before moving or lending memory, so a failed send transfers nothing. If a multi-page transfer fails midway, the kernel must unwind the pages already moved rather than leaving half the range in one process and half in another. If a server exits while holding borrowed memory, the page must return to the sender or the reserved range must be re-backed so the caller can recover. These aren’t use-after-free bugs in the traditional C sense—they’re ownership protocol bugs. Rust can help represent the states, but the states themselves are runtime facts about processes, queues, and page tables. The memory-safety guarantee depends on the kernel state machine being correct.

Prior Art and Where Palladium Fits

None of this is entirely novel territory. Capability microkernels—seL4 and the L4 family—govern cross-domain access with capabilities; seL4 has a formally verified C implementation and represents the state of the art in machine-checked OS security proofs. Singularity, Microsoft Research’s experimental OS, went in nearly the opposite direction: mutually distrusting processes shared a single address space, with ownership enforced by a language and verifier rather than by hardware isolation. Mach and related systems already move memory regions between tasks during IPC. What Palladium adds is an explicit borrow-and-return discipline attached to that movement—distinguishing a permanent move from a temporary lease, validating the return at the page-table level, and expressing the four shapes as a typed ABI that maps directly to Rust’s ownership vocabulary.

Memory Safety as a Stack

The assumption we started with was simple: write the OS in Rust and get a memory-safe OS. Building Palladium made that statement more precise. Rust makes code inside a process dramatically safer—it makes ownership visible, narrows the blast radius of unsafe code, improves cleanup, and gives API designers better tools. Those wins are real and worth building on. But the hard OS problems aren’t only local aliasing and lifetime bugs. They’re structural: address-space transitions, page granularity, queue admission, blocking intervals, hostile peers, process death, and future multicore revocation. At that layer, memory safety isn’t delivered by the compiler. It’s built from language rules, kernel state machines, page tables, runtime validation, scheduler behavior, and hardware assumptions—a stack, not a single property. Rust gave Palladium the vocabulary of ownership. The microkernel carries that vocabulary to the places Rust can’t see.

Ken Carpenter is cofounder and CTO of Foundation. He leads the development of KeyOS, the security-focused, Rust-based microkernel operating system that powers Passport Prime. Ken oversees hardware architecture, organizational security, and the KeyOS SDK, which turns Passport Prime into a programmable security platform for third-party developers. He is a prominent voice in open-source technology and applied cryptography, and is helping to usher in post-quantum cryptography for Bitcoin, AI, and the broader security industry.

You Just Read

Where Rust Ends and the Kernel Begins

© 2026 Copyright held by the owner/author(s).