GitHub - cheapsteak/duh: duh — du, honest. APFS-clone-aware disk usage analyzer for macOS: measures what deleting actually frees

GitHub

6 min read Original article ↗

du sums per-file sizes. On APFS, clones make that a lie: a directory can "contain" 20 GB and free 3 MB when you delete it. duh is a macOS disk usage analyzer that reports what deleting actually frees.

The problem

On APFS, du, Finder, and GUI analyzers like DaisyDisk sum per-file sizes. But APFS clones — created by clonefile(2), i.e. cp -c, and used heavily by tools like pnpm, uv, Postgres's FILE_COPY clone-based database branching, and git-worktree-heavy setups — report their full size on every clone while sharing physical blocks on disk. A directory tree full of clones can overstate its real disk cost by 10–100x.

Hardlinks have the mirror problem: several paths, one set of blocks.

The only ground truth is the volume's allocated-block count (what df reports) — and df attributes nothing to folders. duh bridges that gap.

What duh does

  • Scans a directory tree into a SQLite database. The design is memory-bounded and streaming; trees of ~4 million files are fine.
  • Detects clone families by APFS clone ID (via a getattrlist binding for ATTR_CMNEXT_CLONEID) and hardlink families by inode.
  • Computes freeable(dir) — exactly what rm -rf dir would return to df. A family's blocks are credited once, at the lowest common ancestor of its members. Families with members outside the directory credit nothing to it: deleting your copy doesn't free blocks something else still references.
  • Reports locked_here / clusters — "delete-together" groups: space that no single child of a directory can free alone, but that frees if you delete the sibling set jointly.

The treemap UI

opens http://127.0.0.1:7777/ — a zoomable treemap with three size modes:

  • Freeable — marginal cost; what deleting this subtree returns to df
  • Allocated — physical blocks, double-counting clones (the du -A-ish view)
  • Logical — apparent file sizes (the classic du/Finder view)

Clone families that span siblings show up as "shared across N children" rather than being silently attributed to one of them.

Sharing

The Share button in the treemap header (in duh serve) uploads a snapshot of the currently-viewed subtree to a secret (unlisted) GitHub gist on your own account, and copies a link to the static viewer at https://cheapsteak.github.io/duh/v/ that renders it as an explorable treemap. This requires the gh CLI installed and signed in (gh auth login) — if it isn't, the button shows a clear error instead of silently failing.

"Secret" means unlisted and effectively unguessable (a random id), not access-controlled: anyone who has the link can view it, so treat the link itself as the secret, same as you would a screenshot. You can delete a shared snapshot at any time from https://gist.github.com (find it in your gists list, or via gh gist delete <id>).

Install

Download the prebuilt binary — macOS, universal (Apple Silicon + Intel). Latest release: https://github.com/cheapsteak/duh/releases/latest.

curl -L -o duh.tar.gz https://github.com/cheapsteak/duh/releases/download/v3.1.0/duh-3.1.0-macos-universal.tar.gz
tar -xzf duh.tar.gz
xattr -d com.apple.quarantine ./duh    # unsigned binary — clears the Gatekeeper block
./duh --version

The xattr step is needed because the binary isn't code-signed or notarized (that requires a paid Apple Developer account); without it macOS blocks it with "cannot be opened because the developer cannot be verified." Move it onto your PATH (e.g. mv duh /usr/local/bin/) to run it as just duh.

Or build from source — needs the Rust toolchain (brew install rust):

git clone https://github.com/cheapsteak/duh && cd duh
cargo install --path .        # or: cargo build --release → ./target/release/duh

Requires macOS on APFS. The Share feature additionally needs the gh CLI signed in (gh auth login).

Quickstart

# 1. Verify clone detection works on your filesystem
duh selftest

# 2. Scan (this can take a while on a large home directory)
duh scan ~

# 3. Ask questions
duh freeable ~/some/dir     # what would rm -rf actually free?
duh top --under ~ -d 2      # biggest directories
duh clones                  # clone families ranked by apparent "waste"
duh clusters                # delete-together groups

# 4. Or explore visually
duh serve

Useful knobs:

  • --exclude NAME adds a directory name to the skip list; a default list already skips regenerable trees (node_modules, .venv, __pycache__, .git/objects, Rust target, etc.). --include NAME removes a default; --no-default-excludes disables the list. duh excluded shows what was skipped and how big it was.
  • --min-free GIB aborts a scan if the volume's free space drops below the threshold (default 3 GiB) — a guard against the scan's own DB growth on a nearly-full disk.
  • The database lives at ~/.local/share/duh/scan.db; override with the DUH_DB environment variable or --db.
  • One database holds at most one scan per subtree: scanning a root that contains, equals, or is contained by an already-scanned root is refused, because indexing the same files twice poisons clone/hardlink analysis and wrecks freeable. Use --rescan to redo the same root, --replace-overlapping to evict the conflicting scan(s), or a separate database. Sibling roots coexist fine.

Other subcommands: marginal PATH, file PATH, stats, and sql (opens the database in sqlite3 with convenience views).

Requirements

  • macOS on APFS (the tool exits immediately on other platforms; clone detection is APFS-specific).
  • The duh binary. Build it with cargo install --path . (or cargo build --release and run ./target/release/duh) — a Rust toolchain is needed only to build, not to run. Python is not required to use duh.
  • No network access needed. The serve treemap UI ships embedded in the binary and runs fully offline.

Gotchas

  • ATTR_CMNEXT_CLONEID is 0x100, not the documented 0x40. Apple's headers/docs suggest 0x40, but empirically on current macOS the attribute is returned for bit 0x100. The constant in src/attrs.rs is the empirically correct one; the self-test (duh selftest) verifies it end-to-end by creating a real clone and a real copy and checking their clone IDs.
  • Sparse files (e.g. a Docker Docker.raw) are counted by allocated blocks, not logical size — a "64 GB" sparse image that occupies 9 GB counts as 9 GB. This is correct for "what would deleting free" but differs from what Finder shows.
  • The SQLite file never shrinks between rescans. Deleted rows leave free pages that get reused, but the file itself only grows. If you want the space back, delete the DB (rm ~/.local/share/duh/scan.db) or run VACUUM via duh sql.
  • Freeable numbers are relative to the scanned root: a clone family member outside anything you scanned can't be seen, so its family may look fully-freeable when it isn't. Scan the broadest root you care about.

Reference implementation

duh began as a single Python script. That original implementation now lives at reference/duh-py, frozen as the parity oracle: the Rust port was built against it test-for-test. The black-box suite in blackbox/ runs against either implementation (set DUH_BIN), and blackbox/test_db_parity.py scans the same tree with both and diffs the resulting databases. The oracle-baseline git tag marks the frozen state the port was validated against.

The one behavioral addition in the Rust binary: duh serve binds localhost only and rejects requests whose Host header isn't a loopback name, guarding against DNS-rebinding from a browser.

License

MIT — see LICENSE.