Show HN: Hashsync – Deduplicate SSH streams against the receiver's files
github.comOn a recent trip, while stuck on spotty hotel Wi-Fi and roaming 5G, I got fed up with how much bandwidth commands like rsync and nix-copy-closure waste.
rsync's delta transfer compares each file only with the file of the same name on the destination (or, with --fuzzy, a file in the same directory), so renamed, moved and copied files are sent in full.
nix-copy-closure can be worse: a rebuilt package gets a new store path, so the whole package is sent again, even when most of its contents haven't changed.
So I created hashsync, which runs a command over SSH and deduplicates the command's output, input, or both against an index of the receiving host's files.
As an example, for my last NixOS update, nix-copy-closure would have sent 1.7 GiB with --gzip, but hashsync took it down to 0.7 GiB instead. On some of my backups, I've seen up to 2x-10x less data and time to transfer compared to rsync with compression.
Two things make hashsync unique:
1. It searches at every byte offset of the stream, whatever its format, for blocks of the indexed files. In effect, it works like rsync's algorithm, but across every indexed file.
2. It's a transparent pipe, so it works with rsync, tar, or anything else you'd send through ssh.
See the repo's README for more info.
Feel free to try it out and let me know if you have any questions!
what happens if the receiver's index is stale and one of the files changes during the transfer? do you verify the final reconstructed stream end to end before giving it back to the command or is correctness relying on the block hashes?
Wow, I don't know what happened... your comment was posted 11 days ago but it only became visible for me now!
> do you verify the final reconstructed stream end to end before giving it back to the command
No, I excluded that design from consideration, for two reasons:
1. It would make everything much slower given the use cases hashsync is intended to be used for. A command pipeline finishes much faster if the hashsync's output starts to be emitted as soon as it's received.
2. Even if performance wasn't an issue, it wouldn't work for rsync, since it requires back-and-forth communication between the rsync instances that run on each host. If hashsync kept the output to itself until it had received the entire data stream, rsync would hang.
So correctness does rely on the block hashes, which would only be a problem if there was a hash collision, which is practically impossible given that hashsync uses 256-bit cryptographically-strong hashes (Blake3 or optionally, SHA-256).
If the index is outdated or an external process races to modify a file while hashsync reads it, it doesn't cause a correctness issue: the block hash is always recomputed after the block is read into memory, before it's transmitted as a reference. If the hash doesn't match anymore, hashsync will discard that block and try to use another block with the same hash -- or if there is no other copy that was indexed, hashsync will transfer the block's (compressed) contents rather than a reference to the block.
In reality, it may not even send the entire block's contents: e.g. if 10 bytes later hashsync again finds a block in the index with a matching hash, it may send those 10 bytes as a literal value, followed by a reference to the new block's hash.