How we built our startup (it's all about balance)
whimsical.coGreat write up, thanks!
I hadn't heard of Terraform before, so thanks for the introduction. I have recently been attempting to familiarize myself better with systems infrastructure, and it looks interesting. I'll have to do some reading on it.
I have a question. You mentioned that Whimsical's back-end, "mostly serves as a lightweight layer between the client and the data store." Does this mean that you do most of the data munging client side so that once it arrives server side, most of the work is already done? And if so, what do the trade-offs look like in such a scenario? If I'm understanding correctly, this means the client is fatter, but there's less load on the server, meaning cheaper infrastructure costs. Is this a correct way of looking at it?
I'm just a junior dev trying to gain a better understanding of large systems, so forgive me for any ignorance or misunderstanding.
Indeed, most of the application logic is in client side. In fact it's built in a way that it could function as a standalone application without backend. Currently it doesn't persist the data locally but that could be easily added.
The backend serves as a fairly generic datastore with an API that allows data to be easily synced back and forth. It implements a logic (again non-specific to Whimsical) for how to handle concurrent updates (basically it implements CRDTs). I believe it's pretty similar to Google's Cloud Firestore - we might have chosen that if it was out of beta and we didn't mind the lock-in.
Two main reasons we did this: 1) to be able to easily add offline mode - we don't have it yet but it's fairly easy to add now 2) to minimize effect of network latency on user experience
As for less load on server & cheaper infrastructure costs - you are right, that is another benefit.
I would be very interested to learn how you manage changefeeds, e.g. distributing updates to client apps. I used RethinkDB for this, as every other solution seemed like lots of extra work.
We use combination of Redis pub/sub and Websockets. Each document has a unique key and whenever any of the users/clients make a change to it, it's published via that key. Other clients connect via WebSockets to backend and subscribe to the same key in Redis. The notifications are fairly simple.
The complex part is how we sync actual data from clients to server while ensuring consistency. We basically store all data in a tree format and having a diffing logic how concurrent changes can be applied conflict-free in most cases. I'm considering writing a separate post on that.
Great post, thanks ! What is "red" above "re-frame" ?
It's an extension to re-frame that we built. It addresses some of the (subjective) shortcomings of re-frame, mainly the over-reliance on keywords. We'll be open-sourcing it soon.