GitHub - Barre/ZeroFS: ZeroFS: A log-structured filesystem for S3. ZeroFS serves S3-compatible buckets as POSIX filesystems over NFS and 9P, or as raw block devices over NBD.

ZeroFS serves S3-compatible buckets as POSIX filesystems over NFS and 9P, and as raw block devices over NBD. All three servers run in one userspace process. Data is compressed and encrypted before upload.

Quick Start

apt / dnf (recommended, amd64/arm64)

# Debian / Ubuntu
curl -fsSL https://pkgs.zerofs.net/zerofs.gpg | sudo gpg --dearmor -o /usr/share/keyrings/zerofs.gpg
echo "deb [signed-by=/usr/share/keyrings/zerofs.gpg] https://pkgs.zerofs.net/deb stable main" | sudo tee /etc/apt/sources.list.d/zerofs.list
sudo apt update && sudo apt install zerofs

# Fedora / RHEL / Rocky
curl -fsSL https://pkgs.zerofs.net/zerofs.repo | sudo tee /etc/yum.repos.d/zerofs.repo
sudo dnf install zerofs

Packages also install a systemd service (zerofs.service, disabled by default) and a config skeleton under /etc/zerofs/. Set ZEROFS_PASSWORD and credentials in /etc/zerofs/zerofs.env, the [storage] url in /etc/zerofs/config.toml, then sudo systemctl enable --now zerofs. Details: packaging/README.md.

Install script

curl -sSfL https://sh.zerofs.net | sh

# Pin a release and install without root
curl -sSfL https://sh.zerofs.net | VERSION=v1.2.5 INSTALL_DIR=$HOME/.local/bin sh

Downloads the release tarball, verifies the published SHA-256 checksum, and installs the prebuilt binary: Linux (amd64, arm64), macOS (x86_64, aarch64), FreeBSD (amd64). Full matrix: quickstart.

Docker

docker pull ghcr.io/barre/zerofs:latest

# Generate a starter config on the host ("-" writes to stdout)
docker run --rm ghcr.io/barre/zerofs:latest init - > zerofs.toml

$EDITOR zerofs.toml
docker run --rm -v "$PWD/zerofs.toml:/zerofs.toml" \
  ghcr.io/barre/zerofs:latest run -c /zerofs.toml

The container runs as UID 1001, not root; a bind-mounted cache directory must be writable by UID 1001. To reach the servers from the host, bind addresses to 0.0.0.0 and map a port per enabled server: 2049 (NFS), 5564 (9P), 10809 (NBD).

Running

zerofs init            # Generate zerofs.toml
$EDITOR zerofs.toml    # Set S3 credentials
zerofs run -c zerofs.toml

Testing

• pjdfstest — 8,662 POSIX cases (pjdfstest_nfs), once per protocol: NFS, 9P, FUSE. Per-protocol exclude lists are in .github/.
• xfstests — the standard filesystem regression suite, over NFS, 9P, and FUSE.
• Kernel build — the Linux kernel compiles with make -j$(nproc) on NFS, 9P, and FUSE mounts.
• stress-ng — file-handling stressors run concurrently against live mounts. - ZFS — a ZFS pool on ZeroFS block devices; kernel source extraction, then a scrub.
• Jepsen local-fs — random operation histories against a 9P mount, checked against a reference model (local-fs). A crash mode kills the server mid-run and verifies recovery matches the last fsync. - Jepsen HA — a leader/standby pair over MinIO under a nemesis that kills or pauses nodes; no acknowledged write may be lost, resurrected, or corrupted across failover. The local-fs model checker also runs with failovers injected.

Web UI

[servers.webui]
addresses = ["127.0.0.1:8080"]
uid = 1000  # POSIX identity for file operations from the browser; required
gid = 1000  # Required

The file manager speaks 9P over WebSocket. Drag-and-drop uploads work, including entire folders.

The dashboard streams stats over gRPC-web, plus a file access tracer.

The terminal boots a Linux VM via v86, with the filesystem at /mnt over the same 9P WebSocket. The guest has no network device.

Architecture

The NFS, 9P, and NBD servers and the Web UI share a single filesystem layer. File contents are split into 32 KiB extents; each extent is compressed, encrypted, and packed as a frame into immutable segment objects (up to 256 MiB). Metadata (inodes, directory entries, and one 32-byte pointer per extent) lives in an LSM-tree database on the same object store. architecture documentation.

graph TB
    subgraph "Client Layer"
        NFS[NFS Client]
        P9[9P Client]
        NBD[NBD Client]
        WEB[Web Browser]
    end

    subgraph "ZeroFS Core"
        NFSD[NFS Server]
        P9D[9P Server]
        NBDD[NBD Server]
        WEBUI[Web UI]
        VFS[Virtual Filesystem]
        SEG[Segment Store<br/>file data as compressed, encrypted frames]
        SLATE[LSM tree<br/>metadata + 32-byte extent pointers]
        CACHE[Local Cache]

        NFSD --> VFS
        P9D --> VFS
        NBDD --> VFS
        WEBUI --> VFS
        VFS --> SEG
        VFS --> SLATE
        SEG --> CACHE
        SLATE --> CACHE
    end

    subgraph "Storage Backend"
        SEGOBJ[Immutable segment objects<br/>segments/shard/epoch/counter]
        SSTS[Metadata SSTs + manifest]
        S3[S3 Object Store]

        CACHE --> SEGOBJ
        CACHE --> SSTS
        SEGOBJ --> S3
        SSTS --> S3
    end

    NFS --> NFSD
    P9 --> P9D
    NBD --> NBDD
    WEB --> WEBUI

High Availability

A [replication] section runs a leader and a standby backed by the same bucket; there is no second copy of the data to provision. The standby semi-synchronously replicates acknowledged-but-unflushed writes and takes over in seconds if the leader fails. Writer-epoch fencing prevents split-brain: a deposed leader cannot commit. Design, guarantees, configuration: high availability.

Configuration

TOML with $VAR/${VAR} environment substitution; all referenced variables must be set. [cache], [storage], and [servers] are required. Full option reference: Configuration Guide.

[cache]
dir = "${HOME}/.cache/zerofs"
disk_size_gb = 10.0
memory_size_gb = 1.0  # Optional, defaults to 0.25

[storage]
url = "s3://my-bucket/zerofs-data"
encryption_password = "${ZEROFS_PASSWORD}"

[filesystem]
max_size_gb = 100.0     # Optional; writes past the quota return ENOSPC (default 16 EiB)
compression = "zstd-3"  # Optional: "zstd-{1-22}" (default "zstd-3") or "lz4"
                                                                   [servers.nfs]
addresses = ["127.0.0.1:2049"]

[servers.ninep]
addresses = ["127.0.0.1:5564"]
unix_socket = "/tmp/zerofs.9p.sock"  # Optional

[servers.nbd]
addresses = ["127.0.0.1:10809"]
unix_socket = "/tmp/zerofs.nbd.sock"  # Optional

[servers.rpc]
addresses = ["127.0.0.1:7000"]  # Needed by zerofs checkpoint, flush, monitor, fatrace, otrace

[aws]
access_key_id = "${AWS_ACCESS_KEY_ID}"
secret_access_key = "${AWS_SECRET_ACCESS_KEY}"
# endpoint = "https://s3.us-east-1.amazonaws.com"  # For S3-compatible services
# default_region = "us-east-1"
# allow_http = "true"  # For non-HTTPS endpoints (e.g., MinIO)
# conditional_put = "redis://localhost:6379"  # For stores without conditional-put support

Backends

url = "s3://bucket/path"        # + [aws] credentials
url = "azure://container/path"  # + [azure] storage_account_name / storage_account_key
url = "gs://bucket/path"        # + [gcp] service_account, or ambient ADC on GCP VMs/GKE
url = "file:///path/to/storage" # Local disk; no credentials

Further schemes (s3a://, abfs://, host-routed https://, memory://): Configuration Guide.

ZeroFS requires conditional writes (put-if-not-exists) for fencing. AWS S3 supports this natively; for stores that don't, set conditional_put to a Redis URL.

An optional storage_class under [storage] is passed verbatim to the backend (S3 x-amz-storage-class, GCS x-goog-storage-class, Azure x-ms-access-tier). Use a hot, standard-access class: archive tiers render the volume unusable, and infrequent-access tiers charge retrieval on ZeroFS's constant reads, usually costing more.

Mounting

Over 9P, fsync returns only after data reaches stable storage; NFS COMMIT semantics let fsync return before that. If you depend on fsync durability, use a 9P-based mount.

zerofs mount (recommended on Linux)

zerofs mount 127.0.0.1:5564 /mnt/zerofs        # TCP
zerofs mount /tmp/zerofs.9p.sock /mnt/zerofs   # Unix socket

Kernel 9P client

mount -t 9p -o trans=tcp,port=5564,version=9p2000.L,cache=mmap,access=user 127.0.0.1 /mnt/9p
# Unix socket
mount -t 9p -o trans=unix,version=9p2000.L,cache=mmap,access=user /tmp/zerofs.9p.sock /mnt/9p

NFS

ZeroFS reports NFS writes as stable while they are buffered; tested clients (macOS, Linux) do not send COMMIT on fsync. Use a 9P mount where fsync durability matters.

# macOS
mount -t nfs -o async,nolocks,rsize=1048576,wsize=1048576,tcp,port=2049,mountport=2049,hard 127.0.0.1:/ mnt
# Linux
mount -t nfs -o async,nolock,rsize=1048576,wsize=1048576,tcp,port=2049,mountport=2049,hard 127.0.0.1:/ /mnt

Mount options, persistent mounts, Windows: NFS access.

NBD Block Devices

Device files in a .nbd directory attach as raw block devices:

# Create devices through any file mount
mkdir -p /mnt/zerofs/.nbd
truncate -s 1G /mnt/zerofs/.nbd/device1

# Connect (recommended: -persist, -timeout 600 for S3 latency, -connections 4)
nbd-client 127.0.0.1 10809 /dev/nbd0 -N device1 -persist -timeout 600 -connections 4
# Unix socket
nbd-client -unix /tmp/zerofs.nbd.sock /dev/nbd1 -N device1 -persist -timeout 600 -connections 4

mkfs.ext4 /dev/nbd0
# or
zpool create mypool /dev/nbd0

The handshake advertises FLUSH, FUA, and multi-connection support. FLUSH and FUA replies return only after data is durable, and a FLUSH on any connection covers all connections, so write barriers hold for ZFS pools and databases. Details: NBD devices.

New device files are picked up at runtime. Sizes are fixed at creation: to resize, disconnect, delete, and recreate. To remove a device, disconnect the client (nbd-client -d /dev/nbd0), then rm the file.

TRIM

fstrim /mnt/block                    # Manual
mount -o discard /dev/nbd0 /mnt/block  # Automatic (filesystems)
zpool set autotrim=on mypool         # Automatic (ZFS)

TRIM deletes extent pointers and debits each segment's live-byte counter; a GC pass every 60 seconds deletes dead segments and repacks fragmented ones, reclaiming the space in S3.

Limits

• Maximum file size: 16 EiB
• Maximum filesystem size: 16 EiB
• Files over the filesystem lifespan: 2^64
• Hardlinks per file: 2^32

Format limits (64-bit inode and size fields, 32 KiB extents), not tested ones; provider limits and storage cost take effect first. See architecture.

Licensing

Dual-licensed under the GNU AGPL v3 (fully featured, for open source use) and a commercial license.