Ask HN: Learning practical system design
Hi all. In the world of AI I might be late to the party, but hey, nothing ventured, nothing gained.
I have worked as a programmer for some years, but I still feel absolutely incapable of designing a highly scalable system and in a way that’s cost effective (without wasting money unnecessarily).
I have read system design books and designing data intensive applications. They are useful but I don’t feel like I can “problem solve” effectively in this domain.
For instance, I got asked in an interview how to scale a web crawler and I froze. Ask me how google can give an AI overview so quickly and I wouldn’t know where to start.
Most of what I do at work is adding new features and fixing bugs. Companies often dictate things like “we’re a Kubernetes, Postgres and redis shop”.
For others that have had to learn these skills without learning from someone at your company, how did you do it? And how do you approach solving these problems?
I understand this is a vague question but I’d appreciate any sort of pointers. Even if you’ll be telling me “the obvious”, that will just help me to trust that advice more (I struggle to trust my own judgement).
Thanks.
Effective system design comes from understanding the constraints of the problem, understanding the constraints of technologies you could use to solve the problem, and understanding what parts of your solution needs to change as those constraints change.
For instance, a web crawler. Establish your boundaries right away. You can't crawl the entire web right out the gate, so you need to decide where to stop. 1M pages? 1B? Does your crawler run continuously, or in batches, on a schedule? Are you concerned with avoiding crawling duplicate links? How often do you want to revisit links you've seen before? Are you going to limit page sizes you'll keep? Are you keeping images and other files, or only HTML content? Do you need to be able to execute JavaScript? How do you want to access the results: search, machine learning, analytics, etc? How many requests for those results do you expect in a given time frame? What's the maximum time you'd like to wait for a response?
All of these questions will affect your system design.
Take those answers and do some napkin math to figure out the scale you're working with.
If you are crawling only 1M pages and keeping only the HTML of pages 50kB in size you can easily use any database off the shelf with little tuning, for pretty much any use case (modulo response time requirements). For allowing larger files and images you'll probably want to reach for commercial object storage (unless you're a whizzkid file server guy), and if you're crawling a billion links you'll likely need to think carefully about horizontal scale and partitioning, and if you need to provide search or analytical processing you'll likely want to bring in specialized data stores, which means setting up some pipes to connect them through replication or batch processing...
It goes on and on like that. You need a broad knowledge of the capabilities and limitations of different softwares across various categories to understand what your options are and when the complication of adding them would be warranted. I find it helps to understand the historical context under which they were first developed, so that you can understand the problem they were trying to solve.
In general, I've had the misfortune of not being the dumbest guy in the room at any given time, so I've not received much in the way of mentorship. I've had to mostly learn by doing, which gives me a particular bias towards building over buying and avoiding bringing in new software dependencies where I can help it.
I encourage you to learn by doing as well, even if you stop around the first or second roadblock, or about the time you start needing to spend your own money to go further. Maybe try implementing things locally in a way that doesn't let you do more from your personal machine per se, but instructs you in the complication of setting up e.g. kafka or elastic search, and their benefits and tradeoffs.
Think hard about how you would monitor the system to make sure it's working well, think about the dashboard you'd want to see to assure you of the health of the system. Think about where the bottlenecks to performance would be. Think about what you'd change to ensure it stays up 99.999% of the time, even when you're deploying changes to it or scaling up the database. Think about what you'd need to do to ensure that you never lose data.
And then read other people's experiences, but remember to take a grain of salt with you. People like to talk a big game, you'll hear people say that tool XYZ is the ONLY choice for a serious project and... that may or may not be true. Depends on the scale.
But usually in system design interviews people want you to whip out all the keywords and show you know why you're using them and what they're for. So when prepping for those, don't be afraid to say "streaming CDC events with debezium and kafka" instead of "batch job loading new records from postgres" or w/e, as long as it's applicable and you can justify it.
System design interviews also tend to revolve around a few categories of problems (web crawling/search, Google docs, building social media feeds, etc etc) so reading other people's answers for those will start giving you the flavor.
I give the system design interviews at my company (well, now I mostly train other people how to do them) so usually I look for people to ask this constraint questions, show breadth of knowledge, reveal where they're strongest by going deep on a few topics, and the more senior they are the more I want them to lead the conversation and ask me questions rather than the other way around. I don't really expect people to come in and deliver a rote-memorized google-scale answer off the top of their head, and tbh I'd be really suspicious if they did.