Wal3: A Write-Ahead Log for Chroma, Built on Object Storage
trychroma.com> In short, we believe that systems design is about contracts between reality and reality as we see it; reality as we see it is just a fiction. We believe systems exist in two forms: their actual reality and our fictional understanding of them. To us, the programmers, the system's reality is a fiction; we cannot see it, we can only observe it through additional SRE observability and monitoring mechanisms. To the system, this observability is a fiction. Whether due to tooling problems or just measurement error, the fiction presented via metrics, logs, and traces resembles the system itself, but is decidedly not the system itself.
this is very McLuhan/systemantics of you! all abstractions are leaky, but some abstractions let you look at the leaks.
TIL about setsums - one wonders if `fn setsum([String]) -> Digest` works then "nested setsums" must also work for very large scales.
one thing i missed from this post, which otherwise would score perfect marks for a technology introduciton, is benchmarks vs your comparisons on warpstream and friends.
Can you elaborate on what you mean by "nesting" setsums? It's an associative and commutative construction, so you can run e.g. map-reduce to map every parquet file to a setsum and then reduce to roll them up.
You will always find a copy of McLuhan on the Chroma teams bookshelf !
Is there a particular book of McLuhan that ya'll recommend?
Is this deeply coupled with Chroma or something we can fork out for other applications? I would love to explore this for lightweight change data capture on S3.
Absolutely! We took care during the build to address chroma’s needs, but I believe the only requirement is chroma storage. Long term it’ll appear on crates.io as a standalone package.
Always exciting to see Robert Escriva's next database project. Building on top of the incredible engineering of S3 is clever, I'm definitely going to be looking at and learning from the implementation in wal3.
Looking forward to the benchmarks on using S3 for piece and the design of the scale out architecture.
Hi,
I'm Robert, one of the authors of wal3. Here to answer any questions or explain more about our engineering philosophy.
Happy hacking, Robert