Why ZFS is actually cool

TL;DR: fsync promises two things, durability and ordering, and I only need ordering. ext4 gives both or neither: every fsync commits the journal and flushes the disk cache. ZFS with sync=disabled gives ordering alone: transaction groups land in order, and a crash loses the last tens of seconds, not a database. My two HDDs are 2.7% busy instead of 13%, and a write takes 2 ms instead of 16.

I thought ZFS was overhyped#

Back in 2016 I ran Proxmox with ZFS on one node like everyone else, then decided a single node doesn't need it: ext4 and btrfs are good enough. Ten years later fsync on ext4 turned out to be too expensive, and ZFS is one of the few filesystems that can do anything about it. So I gave it another go, and on the way changed my mind about datasets and snapshots too.

fsync on ext4 is too expensive#

Take my box as the example: about ten services, most of them on SQLite, on two HDDs in mdadm RAID1 with ext4 on top. I took a tradeoff on purpose: it's a homelab with no payment data, and most of the state mirrors something external that I can upload again, so rolling back a minute after a power cut is fine. The box had also seen zero outages in 370 days of uptime. So I set commit=60, which commits the ext4 journal once a minute instead of every five seconds, and expected fewer writes.

Disk utilization stayed high anyway, because fsync doesn't give a shit about commit=60. fsync writes out the file's own data, forces a journal commit of the whole running transaction immediately, and then flushes the disk cache on both mirrored disks with everything anyone has written into it. On a busy HDD the flush queues behind everybody else's in-flight writes, so an fsync of one byte can end up waiting on 3 GB of I/O. fast_commit wouldn't save me either: it shrinks the commit, but every fsync still ends with that flush.

	if (journal->j_flags & JBD2_BARRIER &&
	    !jbd2_trans_will_send_data_barrier(journal, commit_tid))
		*needs_barrier = true;

	return ext4_fc_commit(journal, commit_tid);

issue_flush:
	if (needs_barrier) {
		err = blkdev_issue_flush(inode->i_sb->s_bdev);
		if (!ret)
			ret = err;
	}

	if (journal->j_flags & JBD2_BARRIER &&
	    !jbd2_has_feature_async_commit(journal))
		write_flags |= REQ_PREFLUSH | REQ_FUA;

	bh_submit(bh, write_flags, journal_end_buffer_io_sync);

static void ext4_fc_submit_bh(struct super_block *sb, bool is_tail)
{
	blk_opf_t write_flags = JBD2_JOURNAL_REQ_FLAGS;
	struct buffer_head *bh = EXT4_SB(sb)->s_fc_bh;

	/* Add REQ_FUA | REQ_PREFLUSH only its tail */
	if (test_opt(sb, BARRIER) && is_tail)
		write_flags |= REQ_FUA | REQ_PREFLUSH;

I timed 200 log writes of 1500 bytes, each followed by fdatasync:

It hurts the other side too. The 3 GB write took 59.8 s with the fsyncs running and 42.5 s without them, so fsync from one process slowed its neighbour's write by 40%.

Over 22 hours one disk flushed its cache 10-15 times a second, 20-30 in bursts, and spent up to a quarter of every second inside a flush.

Ordering, not durability#

Unix fsync promises two things: durability and ordering. I only need ordering. An app writes A, syncs, writes B, syncs. After a crash the disk holds both, or neither, or A alone, and never B without A. That keeps the app's checkpointing logic correct. Losing both A and B is fine, since I already accepted losing a minute. Same with files: people abuse write + fsync + rename to publish a file atomically, when in most cases write + rename is enough if the filesystem orders the data before the rename.

Why batching fsync in userspace doesn't help#

The obvious fix is to patch every app, and that means a fork, a rebuild and WAL settings for each one. That's a pain, and I wanted one fix for the whole box.

Making fsync a no-op on ext4 or XFS, with eatmydata or SystemCallFilter=~@sync, loses ordering across files. SQLite's WAL can then land on disk after the database pages it refers to. Crash in between, and the whole database is broken, which is much worse than being a minute behind.

I thought it would be nice to batch those fsyncs instead, so the box pays one flush per window instead of one per fsync. Without searching any further I hacked up a small prototype: a seccomp user notification filter intercepts fsync and fdatasync in every app, and a daemon holds the calls for a window and releases them together.

The daemon has two choices when an app calls fsync, and both suck. If the daemon acks the fsync immediately and flushes later, the app trusts writes that never landed after a crash. I might as well use eatmydata at this point.

If the daemon holds the caller until the window ends, a single-threaded app can't issue a second fsync until the first returns. So each app puts one call into each batch and waits out the whole window. I get fewer flushes because the apps sit and wait for them.

I tested it on ext4 over a loop device on a FUSE file that sleeps 10 ms on every flush:

Four writers get nothing from the window because the kernel already batches concurrent fsyncs: everyone waiting on the same journal transaction is released by one commit and one flush. And when the daemon dies, every fsync in every intercepted process returns ENOSYS, which for Postgres is a PANIC. BADBADNOTGOOD. Batching pays off only when apps don't wait for the flush, which is what ZFS's sync=disabled does. So the fix belongs in the filesystem. And I wasn't the first to get there — people have been working on this since around 2008, see the links at the end.

sync=disabled is a free write barrier#

ZFS commits transaction groups (txgs) in order and finishes each one by writing the uberblock. A txg writes into free space, so nothing the current uberblock points to is overwritten, and the new tree becomes visible only when its uberblock lands after a cache flush. Die before that, and the previous tree is still there intact. After a crash the disk holds a prefix of history. With sync=disabled, fsync returns immediately. A crash loses up to three txgs, tens of seconds on these disks, and the pool comes back consistent.

btrfs and bcachefs are copy-on-write too. bcachefs has options to delay or disable journal flushes, and btrfs has no first-class knob for it.

Copy-on-write also plays nicely with spinning rust: a txg takes logically random writes and lays them down as one sequential run. (GOD I LOVE LSM TREES NEED TO WRITE A POST ABOUT THEM) Btw, bcachefs takes this all the way: the whole disk is an append-only log with a garbage collector. I think it's superior to ZFS, but I'll wait for the Rust transition before I trust it with my data.

The costs of CoW are fragmentation and write amplification from recordsize. Update 4K, write a whole 16K block.

What changed on the disks#

At the end of October I rebooted the box into a ZFS mirror with sync=disabled on every dataset. The table covers both disks during quiet hours and compares the two days before the switch on mdadm + ext4 with the day after on ZFS. The latency rows compare the week before the switch with the three days after:

Utilization is 5x lower. The ZIL counters stayed at zero over 22.7 hours, which makes sense: with sync=disabled nothing goes through the ZIL.

The same two metrics with a day on ext4 laid over a day on ZFS, so the daily backup and the other scheduled jobs line up:

A dataset per workload#

The pool and root have ashift=12, compression=zstd, atime=off, xattr=sa, acltype=posixacl, dnodesize=auto and sync=disabled, and every dataset inherits them, except mail and git, below.

Most of these are about not paying extra seeks on an HDD: atime=off skips a write per read, and xattr=sa with dnodesize=auto keeps extended attributes and POSIX ACLs inside the dnode instead of in a separate hidden object, so reading metadata is one I/O instead of two. ashift=12 just matches the 4K sectors.

Each dataset matches recordsize to how its app writes; state is mounted at /var/lib:

Mail and git keep sync=standard: a pushed commit or an accepted mail is acknowledged to someone else, and losing it after that is not fine. And they have 3 events per day, which is negligible.

There's no dedup, no L2ARC and no SLOG, and ARC is capped at 16 GiB. The pool compresses 2.32x, and /nix takes 21.6 G on disk for 43.5 G of data. JSONL compresses best: 27 G of llm logs take 5 G on disk.

Snapshots are about atomicity#

I thought snapshots were useless. You should have real backups, in several places, and you don't need me to tell you that. Snapshots turn out to be the way to back up a live service: take a snapshot, back up from it, delete it. Without one, I had to stop a service so its database and its files on disk agree. Take an app with metadata in SQLite and blobs on disk, and back it up live. The backup copies the database first and the files after, so it ends up with rows without their blobs and blobs without their rows. Congratulations, you've just orphaned those poor files, and now they're going to get reaped.

Now borgmatic reads from a ZFS snapshot: no service is stopped, the backup got faster, and it costs nothing.

Restoring from backups on NixOS#

A restore from backups is also how the box went from ext4 to ZFS, since there is no in-place conversion.

SSH into the live system. Kexec into an installer. Let disko format the disks. Restore from backups. Reboot.

Sounds too good to be true, right? Well, that's why we have Nix.

It has tradeoffs though#

You have to know what your app is doing with the disk. But the rules are simple:

• Databases. Is it B-tree based, like Postgres, SQLite, or MySQL? Then your recordsize should match the DB page size. If you have a 128K recordsize and a 4K page, every 4K write will rewrite 128K. Buut bigger pages give the compressor a bigger window. That's why state sits at 16K rather than the 8K Postgres page. If it's LSM based then you should be golden.
• Static stuff. Slap a big recordsize on it and forget about tweaking.
• Writes someone else has seen. sync=disabled rolls back the disk, not the world, sadly. If an app told another system about a write, like a mail server accepting a message, the rollback leaves them out of sync. Those datasets keep sync=standard.

Turns out it wasn't overhyped.

Prior art#

What people did about fsync before me:

• Josef Bacik's fsync batching in jbd, then Ric Wheeler's version, which became the ext4 knobs min_batch_time and max_batch_time.
• O_PONIES and auto_da_alloc in 2009, after ext4 and data loss.
• OptFS (SOSP 2013), "Don't stack your log on my log" (2014), and BarrierFS (FAST 2018).
• The 2015 LKML thread "Tux3 Report: How fast can we fsync?".
• Group commit in databases: Postgres commit_delay, MariaDB binlog_commit_wait_*, RocksDB WAL group commit.