Distributed Computing at Airbnb
nerds.airbnb.comI'm always impressed by the technical write-ups that come from Airbnb. I'm also always astounded by the complexity of what's happening on the back-end for what appears, on-the-front-end, to be a relatively straight-forward CRUD-type system.
It is a relatively straight-forward CRUD-type system, it's the amounts of data it deals with coupled with (I am speculating) high volume of traffic what makes it tricky to continue to look like a simple CRUD app from the outside.
The complexity could have been reduced dramatically if they were hosting on a modern dedicated hardware configured for their workloads. Successfully reaching this kind of scale on AWS (or any other massive cluster of exhausted and unreliable virtual generics with limited I/O) requires this kind of software wizardry on the back-end.
It would have been even more complicated if all of AirBnB was running on a cluster of randomly dying first generation iPhones.
Want to avoid layers and layers of software complexity and paying for brainy guys to run it? Pay for a nicer hardware then. Sometimes it's cheaper (and easier), sometimes it's not.
I wish I could upvote this twice. Easily one of the most insightful comments I've seen on HN recently.
This is really true and most every financial services company knows this.
They spend so much money on buying huge machines which makes writing performance intensive code similar to writing regular code - ie if its sucks it'll still perform pretty well.
Think of the biggest financial services companies you know - all of them do this.
The front end is pretty cool too. Check this out, just released:
You need to move away from Posterous. It's closing down in less than 15 days.
We've got something in the works.
I'm curious what other components you use around these services. ZooKeeper for properties? What do your Storm spouts feed from (Kestrel, Kafka, ...), and what do your bolts write to? Are you using something for distributed locking?
The article is a good start, but leaves a lot to the imagination!
Storm Framework links to Hadoop, just an FYI.
Thanks, fixed.