Every ZFS operator has had this conversation. The regularly scheduled monthly ZFS scrub is running and application latency doubles for the next two days. Workloads slow, monitoring alerts, users complain. You move the scrub to a quarterly schedule, then to “before the audit,” then to never. Twelve months later a disk fails, the resilver crawls for a week, and a latent checksum error on a second disk turns a routine replacement into a restore-from-backup incident.
When Maintenance Becomes the Outage
Delaying the scrub is rational from the operator’s chair. The scrub visibly hurts production, and the value it delivers is invisible until the day it saves you. What gets missed is what the pain is telling you. A scrub is a read-only workload that runs at the lowest I/O priority ZFS has. If the lowest-priority class in the scheduler can push your application into unacceptable latency, the pool has no performance headroom at all. The scrub did not create that condition. It measured it.
If the simple act of reading all the data in the storage system causes enough performance impact to degrade protection, that is the canary in the coal mine trying to save you from what comes next.
What a Scrub Actually Does
A scrub walks every allocated block in the pool and verifies its checksum against the block pointer that references it. On a healthy pool it finds nothing. On an unhealthy pool it finds bit rot, failing sectors, and misdirected writes while you still have the redundancy to repair them, which is the entire point.
Since OpenZFS 0.8, scrubs run in two phases. The first phase traverses the pool’s metadata, the tree of indirect blocks and dnodes that describes where everything lives on disk, and builds a sorted list of block ranges in memory. The second phase issues those ranges as large sequential reads. This was a major improvement over the old block-pointer-order scrub, which hammered spinning disks with many small random reads for the entire run. But the first phase is still bound by metadata traversal, and on a fragmented pool full of small blocks, that traversal is a random-read workload no sorting can fix.
Scrub I/O is also explicitly deprioritized. The ZIO scheduler caps in-flight scrub reads (zfs_vdev_scrub_max_active) and throttles the amount of outstanding bytes to be read (zfs_scan_vdev_limit) per VDEV. Operators who feel scrub pain usually reach for these knobs first, and for zpool scrub -p to pause the run during business hours:
# zpool status tank
scan: scrub in progress since Sat Jul 11 02:00:14 2026
18.4T / 214T scanned at 312M/s, 9.1T / 214T issued at 154M/s
0B repaired, 4.25% done, 16 days 02:11:32 to go
A 16-day estimate on a 214TB pool is the number that starts the “can we skip this month” conversation. Turning the tunables down makes the estimate worse. Turning them up makes production worse. The knobs are not the problem. The pool was designed without room for this workload.
Where the Sizing Went Wrong
Pools that suffer during scrubs were almost always sized by two numbers: usable terabytes and price. The workloads that keep a pool healthy never made it into the spreadsheet.
The first failure is capacity. Pools get sized to fit the dataset on day one, and steady-state operation at 85 to 95 percent full becomes normal. ZFS allocation behavior degrades as free space shrinks. The allocator works harder to find contiguous regions, free space fragments, and eventually ZFS resorts to gang blocks, splitting single logical writes across scattered fragments. Every one of those fragments is a block pointer the next scrub has to chase. A full pool is a fragmented pool, and a fragmented pool turns the scrub’s metadata phase into millions of small random reads.
The second failure is IOPS. RAIDZ delivers capacity per dollar, and it delivers roughly the random-read IOPS of a single disk per VDEV, because every disk in the VDEV participates in each logical block read. A 60-drive chassis of 12T drives built as five 12-wide RAIDZ2 VDEVs offers around 500TB usable and roughly 750 random read IOPS. Teams compare that 500TB against their dataset and sign the purchase order. Nobody compares 750 IOPS against production load plus a concurrent metadata traversal, which is exactly what a scrub, a resilver, or a large snapshot destroy will demand.
That is the third failure: maintenance never gets a budget. Sizing models the application workload and stops there. But scrubs, resilvers, snapshot cleanup, and send/receive replication are not optional extras on a production pool. They are the production workload, on a recurring schedule, for the life of the system. A pool sized to barely serve the application is a pool that cannot afford its own upkeep, and the monthly scrub is where that deficit gets collected.
The Economics of Fixing it
The honest fix is more VDEVs, narrower VDEVs, or mirrors, and lower fill ratios. All that costs money, and telling a customer to double their spindle count, in exchange for no additional capacity, is rarely a conversation that goes anywhere. Hardware budgets are real constraints, and design advice that ignores them is useless.
There is an asymmetry worth exploiting instead. Scrub and resilver pain concentrates in metadata: small blocks, scattered across the pool, read in dependency order. Spinning disks are terrible at exactly this and remain excellent at the large sequential reads that dominate the scrub’s second phase. The workload that hurts is a small fraction of the capacity. Move that fraction to media built for that type of workload.
Metadata Offload with a Special VDEV
OpenZFS allocation classes allow a pool to carry a special VDEV, typically a wide mirror of enterprise SSDs or NVMe devices, that receives all pool metadata: indirect blocks, dnodes, spacemaps, even the DDT if you run dedup. Data blocks stay on the spinning disks.
# zpool add tank special mirror nvme0n1 nvme1n1 nvme2n1
The effect on scrub behaviour is direct. The metadata traversal phase, the part that generated the random-read storm on your HDDs, now runs against SSDs at five or six-digit IOPS while the spinning disks handle only the sequential data reads, they are good at. Resilver scanning benefits identically, and so does every production operation that touches metadata: directory listings, stat-heavy workloads, snapshot operations, and space accounting.
The special_small_blocks property extends this to small file data. Blocks at or below the threshold land on the special VDEV alongside the metadata:
# zfs set special_small_blocks=32K tank/vms
Sizing is measurable rather than guesswork. zdb -bb tank breaks down allocated space by block type; sum the non-data categories and you have your current metadata footprint. On typical pools with 128K recordsize, metadata runs well under one percent of allocated space. Plan for growth and for small-block spillover, and a mirrored set of 2TB NVMe devices comfortably serves a several-hundred-terabyte pool. Against the cost of the chassis and drives already in the rack, that is a low single-digit percentage of the hardware budget, purchased specifically to make maintenance affordable.
Two caveats matter. The special VDEV is a top-level VDEV lose it and you lose the entire pool, so mirror it, and use a three or four-way mirror to match the rest of your pools fault tolerance. And allocation classes apply to new writes only. Existing metadata stays on the spinning disks until it is rewritten, so a special VDEV added to an old pool pays off gradually, or immediately if you use the new zfs rewrite feature introduced in OpenZFS 2.4. When the special VDEV fills, allocations spill back to the main pool rather than failing, so undersizing degrades the benefit without creating an outage.
The Scrub is the Fire Drill
A monthly scrub is a rehearsal for the resilver you will eventually have to run, under the same I/O patterns, except the resilver happens while the pool is degraded and every hour of runtime is an hour of exposure to a second failure. A pool that cannot scrub without hurting production cannot resilver safely at all. That is the standard you need to design against.
So, treat scrub pain as the diagnostic it is. Size pools for the application plus its upkeep. Keep fill ratios where the allocator can breathe. Buy IOPS where the workload actually needs them, and recognize that for scrubs and resilvers, the workload that needs them is metadata. A small metadata offload investment is the cheapest headroom you can buy, and it converts the scrub from a monthly incident back into what it was designed to be: background noise that occasionally saves your data.
Getting Back on Track
If your systems are not running regular scrubs, or you would like to investigate what it will take to add a metadata offload VDEV to your pool, Klara can help you safely adjust your existing pool to get the IOPS you need. A consultation to size a new metadata VDEV to handle your existing and future metadata needs, plus determining how much benefit special_small_blocks might provide is a small investment to safeguard your data.
Klara’s ZFS infrastructure support helps identify the root causes of storage bottlenecks and provides practical recommendations to optimize pool design, improve performance, and ensure maintenance tasks like scrubs and resilvers without disrupting production.