Making Atuin sync 32x faster with packfiles
blog.atuin.shAtuin is cool, I've had to disable sync though. It turns out that I don't actually want a global command history, I want a very very local one.
There are some interesting use cases building on top of sync, like "tell me about my agent sessions", but imo that's not useful enough to lose ctrl+r (though as I type I'm realizing it could just not bind Atuin to ctrl+r).
I don't want to be mean, but in my personal experience and projects, I have a kind of rule. If I get a 5% to 50% speedup, then I probably did a good job optimizing. If I get a 2x to 5x speedup then I probably was fixing up a prototype. If I get >10x speedup, I fucked up the first time.
By that logic there is a lot of fucked up software systems in the world we live in..
See it as it is. Atuin is a useful tool that stashes shell history onto SQLite locally. Then it expands to syncing them across machines per identity. They implemented it as simple and useful as it could be for what was needed at the time(shell history is not the scale of some commerce orders). Once they actually realized where the opportunity is to improve it(upon a functional sync already in place), they do and post about it at HN. I don’t even use their sync. I’m happy to learn about packfiles and the idea of optimizing sync like this. Not doing this at first is not a fuck up.. it is the evolution of good practical software.
> By that logic there is a lot of fucked up software systems in the world we live in..
Yes? Yes.
(This is not a jab at Atuin per se. Just a comment on the state of software we are in.)
> By that logic there is a lot of fucked up software systems in the world we live in..
Correct.
> By that logic there is a lot of fucked up software systems in the world we live in..
yes
Hi! I'm the person who implemented packfiles at Atuin.
I totally agree, generally, however, the constraints of software evolve all the time.
Atuin started off as a simple .bash_history replacement, and sync was ultimately relatively simple. We have a WAL-style strategy for sync that sees sync as an AP distributed problem (which is effectively what multi-machine concurrent sync is).
This was implemented originally and alone was very challenging. The implementation made a single RPC to the sync server, then grew into effectively single-threaded batches record uploads.
As you might note from the post, you'll note that the biggest performance improvement weren't packfiles alone, but pipelining. Packfiles make storage cheaper, but the latency reduction came from splitting the endecode part of the pipeline from the actual HTTPS requests.
As you can imagine, pipelineing brings its own array of difficulties and challenges.
Atuin originally solved for correct sync, then it evolved into batched sync, and now it evolved into pipelined sync. Finally, it evolved into sync with packfiles. All of these were solving actual problems people had, and not so much a one-shot attempt at perfection.
Had we strived for sync latency perfection from day-one, we probably wouldn't have E2E encryption. We optimize for the use-case we see in the field, and not for an initial use-case we believe we might see in the future.
Generally software evolves. I'm not even sure whether this is the last architectural performance step---I suspect there are more architectural things we can do to get more performance still, but without a need for that, it's hard to justify spending engineering effort for something nobody needs.
Look at UNIX, as an example. I'd say that's a relatively successful piece of software, but the moment you interact with anything UNIX you end up in ioctl hell, because that's kind of what it evolved into. That's what the users' use-cases evolved into.
It's not perfect, but it solves the problem people have and that's what we aim for!
Surely it's more like, at the start of the project X wasn't even close to being an issue. As the project grows and other things get improved, X suddenly appears as being way too slow
I've seen this dozens of times. If you've never replaced something with something 10x faster you're massively over engineering at the start of a project
Calling the act of not leaving 32x performance bugs in my product "over engineering" is really making me wonder what you think "engineering" is. Now I am being mean, btw.
Hard disagree
Sometimes the naive implementation is fine. It gets you a MVP, and it can take a product far without ever needing an improvement. It lets you build a suite of tests to verify behaviour, and informs you that it's a useful feature in the first place. There's no point spending time on a perfect implementation of something when the slow build might be good enough and it could be something nobody wanted in the first place.
Once it's proved its worth you can then go back and spend some extra effort making it fast, with the safety net of tests to make sure the new (likely more complex and hard to reason about) implementation meets the spec
I do agree. Doing a new "fucked up the first time" in quite a number of ways has often provided the stepping stone toward significant improvements in aspects quite unrelated to the initial task. I can more easily accept the fucking up, having learned that my later fix may provide me with a way to get the >10x uplift on not just this project, but maybe on some of the next projects.
And yes, there IS (ARE) a lot of fucked up software systems loose in the world today... keeps the headlines full, every weekday ending in the letter "y".
Every improvement is welcome. Keep up the good work.
A happy atuin user.
I'll add that at this point I've been programming for 20 years so I have higher standards for myself and the things I make. I understand there are people who don't hold themselves to those standards, and I would love for you all to tell me your product names for future reference.
Not really. Premature optimizations have their own set of problems. It's generally better to keep the design simple and tackle optimization if performance becomes an issue.
I've been using atuin for years without noticing this, so it has definitely not been a significant problem.
So what if you did? Have you never learned from the mistakes you made in V0.1 of your software and then proceeded to make it 10x better?
OK, I had a laugh at the TP-esque quote
This is not just a small performance improvement. This is a paradigm shift in how Atuin does sync.
— lots of numbers on a GPU, August 2026Haha glad you enjoyed it! We had a lot of back and forth in the office to make sure it felt _like_ AI writing without being AI writing as a small joke =)
> We sync many thousands of records in single-digit seconds.
Is that meant to be impressive performance? I'd never heard of Atuin but apparent it syncs your shell history between machines using a Blockchain?
No blockchain there, it's a centralized sync server (you can deploy your own [and you can leave 'sync' disabled too):
https://deepwiki.com/atuinsh/atuin/3.3-synchronization-syste...
Yeah, it's surprising that the numbers are this low, at the end of the day, but sync for Atuin unfortunately isn't equivalent to rsync.
We actually need to decrypt data that comes from the sync server, since everything is end-to-end encrypted and that's actually where the majority of CPU time is spent!
As another commenter mentioned, the data is not distributed, but lives on a central server (which either you host our you use our official one).
Indeed, it seems that a good old fashioned .histfile synced via any VCS would perform better?
Atuin maintaner here!
It definitely would, but the real challenge is twofold:
- Atuin does real-time sync, meaning that anything you do locally gets reflected near-instantly on any other machine you have. - Atuin does end-to-end encryption, which means that we actually need to do an absurdly larger amount of work compared to an rsynced histfile in order to get your history in a format that's locally decrypted.
Certainly, these things are possible with a VCS, but I suspect that meeting these two criteria with an rsynced encrypted file would land on more-or-less the same performance order-of-magnitude!