Discoveries
In the process of getting to this point, we ended up finding one particularly interesting bug, we’ll try to save the war story (sorry for getting your hopes up). Originally, our ID format would only allow 64k unique IDs to be generated per host per second. We had forgotten about this limitation after porting the ids from the old system and got some nasty new log lines. Until we figured out what it was, it looked like a ton of duplicate messages (which would be very bad). We had to update the ID format to now allow over a billion per second per host, which will never happen on a single host, especially since we’re persisting each message. No duplicate messages or duplicate IDs now. D’oh. There is an excellent blog post about IDs here, our new format is very similar.
Sometime in the near future...
Most of the tests only show what a single thread can do on a single queue, where typical use cases use tens to thousands of threads across multiple queues. All of the trials above also show only batches of 100, typically with relatively small messages; latency and request throughput look quite a bit better when only dealing in batches of 1 (at the cost of message throughput). From the million per second and various cases above we found a number of low-hanging fruit (looking at you, RPC, and our p99) and plan to address these before making the next post -- and now we have a sweet testing infrastructure to do so. In the next post we also hope to show results for different message sizes, consistency, batching, and different cluster configurations.
Want to know how it all works? The TL;DR is something similar to view stamped replication via 2PC and a gossip protocol for membership, with synchronous replication to all replicas; meaning that each message is consistent across all replicas. Our unit of sharding is queues, and we automatically spread them around onto different nodes. We use 3 replicas by default for queues, meaning in each of these tests, each message is written to 3 replicas before returning, and we serialize all reads and writes through one of the replicas for each particular queue (i.e. master-slave). At the bottom, we’re running embedded local rocksdb inside of each MQ process for persistent storage of queues and all metadata.
At present, we don’t have comprehensive documentation for all of the nitty-gritty, but we do enjoy chatting about this stuff if you want more details. Feel free to chat us up on the https://gophers.slack.com/messages gopher slack channel.
Want to give IronMQ a whirl for yourself? Recently, we made v3 the default for everyone, sign up and see what you think. Want to load test it or try running on your own machines? Send a note to our award-winning support team (I gave Ukiah one of my Little League trophies so we could say that).
about muah: Reed is currently programming computers at Iron.io. He spends most of his time trying to figure out why machines aren’t doing the things that he told them to do. He thinks this is enough of him talking about himself in third person.