Ask HN: Where can I see many examples of real companies' software architecture?
I want to broaden my horizon regarding how things are solved in the real world. Other than some very high-profile companies (like Netflix, github) and companies that I've worked at, it's hard for me to find easily digestible (20-60 mins) examples of actual working architecture of differently sized companies from different business verticals. You can get a good view on some architectures from AOSA, thought it may not be as focused on web as you're looking for: http://aosabook.org/en/index.html A good example is Scalable Web Architecture and Distributed Systems by Kate Matsudaira: On topic of architecture of open source applications, Wikimedia shares this: It's pretty cool how their Grafana is open https://grafana.wikimedia.org/d/000000002/api-backend-summar... Very cool! At what level of scale might one expect to need what's going on in the "Edge Cluster", as opposed letting all the requests fly right into the app servers? Thanks! I didn't know about the AOSA books. There are some really nice examples of top-level software documentation in them. Wow, those all look fantastic. Thank you for sharing! High Scalability has some good articles around this. Keep in mind that the author of that blog doesn't actually talk to anyone at the company they are writing about, they just collect articles around the internet and public statements and piece it together from that. For example one of the most popular article on that site (which is part of their book now) is the article on Netflix. A lot of that was cribbed directly from my talks, but they never reached out to me to even check it over, and as such missed a lot of nuance and detail, things I didn't cover in my talks. Same thing for the article about reddit -- also cribbed a lot from my talks. It's a fine overview, but light on specifics. I've reached out a few times and some things have been corrected after the fact, but I don't know if the other articles have been reviewed. So my point is, be warned that the articles on that site are not primary sources but are derived from them. Other times, they would directly talk to a single employee, but get skewed or misleading information based entirely just on that one employee's POV. Their post about Tumblr's architecture [1] focused a lot about JVM-based services, HBase, etc which in reality was only ever used for a tiny subset of the backend. The huge section on "Cell Design for Dashboard Inbox" was especially ridiculous: the systems described there were literally a mix of complete vaporware and failed/canceled projects that never even got close to production. As an early Tumblr engineer, I was really upset to read this nonsense. I spent several months of my life working very long hours to successfully scale the existing (PHP/MySQL) dashboard activity feed architecture in 2011-2012. It continued to be used as-is for many years after this interview, with lower latency and much lower cost than the proposed hbase/scala cell replacement. And of course, engineering candidates being interviewed would always ask about this hbase cell architecture thing that they read about in High Scalability... [1] http://highscalability.com/blog/2012/2/13/tumblr-architectur... A popular choice in the real world is known as 'the big ball of mud', e.g. as described at http://www.laputan.org/mud/. It isn't really a choice, it is just what happens due to incentives and changing needs and leaders and politics and technologies inside the company. People assume that software architecture is like building architecture, in some ways it is, but NO ONE has ever showed up to a construction site that was half way done and said "Hey guys the steel framing we ordered has been delayed so please continue building the rest of the building by replacing anything that was original designed for steel beams with bamboo. This is awesome (as well as the ball of mud "paper"). I think the building construction analogy might be similar to a home which has seen multiple remodels. Now imagine a scenario where you have an absentee owner with a lot of money, a permanently staffed architect and a bunch of extremely able, slightly competitive contractors all on staff - each trying to prove their annual salary. The original one story building would quickly become an ten story nightmare of a building. Just another example of Silicon Valley being ahead of its time: The Winchester House is pretty much exactly this. It used to be 5 stories, too, until the 1906 quake gave it a haircut. A better description is that of garden landscaping. You have a reasonable idea of what you want to achieve, and it looks good on paper, but until you have actually walked through it, touched and felt it, you don't know whether it really does what you wanted. And of course, it changes as it matures, and what was great at first can become overgrown and resource-hungry. I like Casey Muratori's view of software architecture being more akin to urban planning than to building architecture. Taking that view seems to alleviate many of the apparent paradoxes. Oh it's definitely a result of choice. Strictly speaking it is lots of choices that in aggregate give the classic result. Some choices by sales, some by engineering, some by management. All doing their best. Each reasonable on a sufficiently small time horizon. A lot of old projects (especially 20 years+) tend to be more or less this. A lot of new projects turn into this within a few months. I came here to share this link too > Other than some very high-profile companies (like Netflix, github) ... it's hard for me to find easily digestible (20-60 mins) examples So I think this doesn't meet your requirements, but I like Tech Dummies Narendra L's YouTube videos [0].
He introduces big tech companies' systems in 30-60min videos and it's not difficult to understand. [0] https://www.youtube.com/playlist?list=PLkQkbY7JNJuBoTemzQfjy... so like.. not to be too cynical but how does he know his representation is correct or at least not misleading? a lot of youtuber content is just made up. Yeah... It's dangerous to accept random YouTubers' content as fact mindlessly. I'm not sure all he says are correct, but at least he uses the target companies' engineer blogs, external articles, and some open-sourced part of systems (and list them in the video's detail section).
His main targets are often big techs like Twitter, Uber, and Netflix, so I guess such documents are often available. Narendra's content is awesome, but I think you're right to be skeptical. His content is more focused on how to answer system design interview questions about how the companies operate. There was an article recently (https://news.ycombinator.com/item?id=30936189) that was basically about how far you can get with a simple architecture. One thing I don't remember explicitly called out, is that most all architectures are grown. There are scarily few situations where starting with the complicated idea is a good idea. Gall's Law : > A complex system that works is invariably found to have evolved from a simple system that worked. The inverse proposition also appears to be true: A complex system designed from scratch never works and cannot be made to work. You have to start over, beginning with a working simple system. So that instead of focusing on the architecture as it-is, we should pay more attention to the evolving stories of software architectures because we can learn more from them. InfoQ's QCon conference frequently has an "architectures you always wondered about", which frequently has good talks. You can find them here: https://www.infoq.com/architecture-design/. I came across this a while back
It is a court document from one of the UK post office horizon IT system scandal.
It has a very detailed review of the system and its history dumbed down to the level a lawyer could (maybe) understand. It stands out because it is quite hard to find examples of this level of detail about such a large scale distributed system which aren't internet / web tech companies. https://www.judiciary.uk/wp-content/uploads/2019/12/bates-v-... Nice! Just browsed through it and was surprised at the great explanation of RDBMS' provided. Excerpt: 21. The main use of a relational database is securely to store large volumes of structured information. The way it does so can be understood as having large numbers (tens, hundreds or even more) of different spreadsheets (which are called tables) and which are linked to one another. Two different tables in a database are linked to one another (in a 'relation') when they both have one or more columns with the same meaning and share values in those columns. I think this is also a important lesson on how to write simple, direct and succinct documentation; no unnecessary fancy "methodologies/processes" required (i am now a fan of Dr. Worden :-). Well worth reading in entirety. PS: Any other overviews of "less sexy" software systems that you can share (eg. Banking, Insurance, Railway Systems etc.)? I am of the opinion that these are the real success stories of the Software Industry. They are battle tested and proven over time. Go to google.com, search "pile of spaghetti", click "images" Gitlab is a very open company and have lots of documentation on their architecture here: https://docs.gitlab.com/ee/development/architecture.html and here: https://about.gitlab.com/handbook/engineering/infrastructure... AWS This is my Architecture video series
https://aws.amazon.com/architecture/this-is-my-architecture/ I'd really want to have that, but for small companies/services that benefited from avoiding the trends to get some competitive advantage. Something like "IT Architecture for the Forbes 500-thousand" A certain amount of information can be gleaned from job descriptions from a company’s careers page. OSINT ;) It's good to note that it depends largely on the company you're looking at. I work for a very large organization (~£6bil in revenue, £700mil in profit last year) and we suffer from the "mud" problem - nothing about our technology stack is particularly special, it's just a hodgepodge of many different technologies that struggle to work together. That's not entirely fair - I work within a very unique solution inside of this firm, but I'm in a very unique position and I'm sad to say that it took a silly amount of hard work just to be able to not work on legacy applications. That being said, the companies you mention (Netflix, Github) work completely differently - they were designed with tech in mind! They probably are much more lean in a technological sense, and don't suffer from enterprise architectural issues that large legacy firms do. I suspect that this inability to move has singlehandedly killed more than one company, though I haven't studied the market to the point that I could really name any. The real kudos has to be given to large companies that existed before the internet and were able to move away from their slow-to-adapt, horribly inefficient legacy systems. I often point out an anecdote from my early e-commerce days in the late 90s where a customer wanted in-store pickup like everyone else they saw on the web doing. But they just had purchased an already outdated (but IBM so nobody got fired) Point Of Sale system for 40 stores which did inventory management as a batch at the close of business each day over Frame Relay lines or even dialup. The concept of a VPN was alien to them. Since they were selling some limited edition high priced items that were allocated to each store, there were often only 1 or 2 at a given location of what would be a popular item. So you can imagine when I explained that the huge investment in the legacy system just a few years before was a big blocker for ‘pick up in store’. I think we had to hardcode something that would remove an item for sale online if there were less than 2 left or some awful hack like that to reduce customer complaints. https://runninginproduction.com is a podcast about software architecture examples. Thanks for linking that. The purpose of each episode is for anyone to walk away having a reasonable understanding of why and how a company built and deployed their app with XYZ technologies without needing to know anything up front. There's over 100 different companies / individuals who were on the show. I tried to make it as efficient as possible to get these details. There's a lot more detail than a few bullet points but it doesn't get super lost in the woods with a million low level details that's specific to 1 company. It's basically an hour or 2 conversation for each episode where we cover everything from building to deploying their app, lessons learned, etc.. look at public org charts, they'll define the architecture. only half joking Conway's law - I don't think it's a joke at all. Like most things in software engineering, it's qualitative and empirical - but also has very strong potential to function as a supporting "first principles" theory for so many things. Conway - "How committees innovate" http://www.melconway.com/Home/pdf/committees.pdf I think this paper has a fantastic corollary in Peter Naur's "Programming as theory building" which triumphantly explores the implications of institutional knowledge in long term software maintenance.
https://pages.cs.wisc.edu/~remzi/Naur.pdf Work at more companies! Lots of great resources here, however, from experience I would say take all public presentations about how things work inside a company with a big grain of salt. They always have a vested interest in advertising successes, and public presentations always focus on some filter of interestingness. You won’t see the important “real world” parts of what’s left out unless you’re part of the organization. Slideshare can give you some insights from various companies, most tech presentations discuss something around their architecture! I was developing some architecture training recently and had this very same question. It’s not easy to find realistic architectures. The best I found was the German contact tracing app — Corona Warn App. It was done by a group of consultancies in collaboration with the German govt, and went from inception to launch in around fifty days — largely if not totally open source. Here’s the repo that has all the architecture in: https://github.com/corona-warn-app/cwa-documentation It’s got full git history so you can see it evolve over time, along with the implementations (also on Github). There’s a pretty fascinating short talk by one of the people who led the project on youtube too — more about the process side though: https://youtu.be/5y1sHSkPWRg?t=1770 Based on my experience, write down the names of some services and languages on slips of paper then draw a few from a hat. In the real world [of software], things are solved by choosing the tech with the lowest barrier to entry, not reading any documentation, getting a minimum-not-quite-viable-proof-of-concept working in a development environment, then making that production, over-working a select few to keep it running, and a lot of crossed fingers and heads in sand. The only thing you'll learn from different verticals and sizes is how size and scope have no correlation to how things are built or whether they work well. The interesting part is how larger scale makes things fail more often, and the response to increased failure can either be running around with your hair on fire for years, or a solid firefighting team, or actually teaching teams not to build products that catch on fire. The only way to get the last one is by focusing on people, not technology. Twitch recently open-sourced all their software. Also Patreon and Microsoft (Bing). I’m sure there are other lesser-known ones that might be interesting too. Oh and I’m pretty sure I’ve seen GitHub Enterprise too. You forgot to add unwillingly, but otherwise correct https://HighScalability.com is excellent Thoughtworks keeps a "technology radar" that I have found very interesting. I won't post the "whys" but it's worth looking for upcoming components and tech that they are seeing used more in consulting. https://www.thoughtworks.com/radar/techniques I thought this was really useful in getting a quick overview of a variety of systems in the real world, even though the book itself is designed to answer interview questions. https://www.amazon.com/System-Design-Interview-Insiders-Guid... Came here to say the same. System Design Interview book has digestible level of info on what the OP is asking (I think :-) ) https://eng.uber.com/
has great articles covering many topics Wow, thanks for this link. At a glance, there are tons of great and interesting articles to view. I am saddened by the fact that there isn't more content on self-driving vehicles. NSA security architecture https://media.defense.gov/2022/Mar/01/2002947139/-1/-1/0/CTR... All of our Sourcegraph docs are public, including our architecture overviews and a myriad other docs linked from there. https://docs.sourcegraph.com/dev/background-information/arch... Also all of our (Sourcegraph's) code is public, so you can see what the architecture actually looks like implemented in code. https://www.Stackshare.io has many Here is an interactive presentation of a serverless architecture with a Stripe integration: https://app.ilograph.com/demo.ilograph.Ilograph/__overview I'm looking for something similar for design interview practice purposes in my job hunt. All the systems design resources I can find are aimed at L4/L5, where the focus is e.g. on how to implement a rate limiter on a single machine, or at best saying you can distribute it by putting the counters on a cache server. I'm trying for L6 and can identify many of the issues with a L5 design (redundancy, sharding, global latency, hot spots, local batching), but it's hard not to miss the obvious, and to offer practical/realworld solutions, when my day job is embedded compilers and not large scale systems. This is mostly a rant but I appreciate suggestions. For AWS, this might help:
https://aws.amazon.com/architecture/this-is-my-architecture/ It's a wonderful question because Github as a zillion projects, and yet there's nary a way to consistently make sense of the system as a whole. Blobs of code. It's hard to see the systems level. I think there's a startup idea in there. I guess searching YouTube is best for these things. https://martinfowler.com/ but I'm not sure if he touches the real world sometimes, it all feels very academic rather than pragmatic. I wonder if you'll find "good" outcomes though, it seems to most startups or companies bumble their way to an architecture that works for them. It might not be correct but it might be best way to build a company without architecting everything too much up front. Most of MF's articles read like someone extremely intelligent but who last actually wrote code or worked on a real system in the late 90's. What made you think that? Just curious, last time I happened to read anything from Thoughtworks was quite a long time ago. Read the engineering blogs of big companies like Google, Netflix, Dropbox, etc. and especially read papers they publish. Google has a book out now about its software engineering practices too--although it's not specifically on architecture you can glean a ton of info about how Google services work internally from its software processes: https://abseil.io/resources/swe-book AWS has a video series based on this
https://aws.amazon.com/architecture/this-is-my-architecture/ I did my own take on this in a short YouTube series https://youtube.com/playlist?list=PLuuxpDWxclfL5BheoL9JyVd0m... Try and search for SDKs of some large software. Usually those would be programs for creating content - audio, 3d modelling, 2d drawing, etc. Every major vendor has a plugin architecture that quite obviously leaks implementation details. So stuff like Adobe Photoshop, Autodesk 3dsMax, FL Studio. All these have public SDKs that you can download, explore and write plugins for. You can probably think of some more programs that support third party plugins. I'm very interesting in the architecture of systems similar to Amazon SQS. Interestingly I couldn't find much discussion on such systems. I guess it's because SQS is such a typical iceberg system that has sophisticated designs to provide dead simple APIs: having a queue that supports competing consumers and simply scales infinitely (in the eyes of users) with users provisioning capacity is no joke. Looks like it hasn’t been updated for a while though. +1... I haven't read through it in a while, but I read it frequently earlier in my career and have found that really valuable. In the cases where I've been asked about situations that I had no first-hand experience in interviews, it's been helpful to draw on knowledge from that reading. Being able to say, "well, Company X got to scale Y using technique Z" sounds more compelling than taking guesses. I've bumped into this in a job listing a few weeks ago from https://litnerd.com/. https://www.figma.com/file/6rFlomXiaKyE4pVQ9O7Qod/engineerin... This is a dated paper, but it checks many of the boxes you ask for - the original Windows Azure Storage architecture: https://sigops.org/sosp/sosp11/current/2011-Cascais/printabl... For Azure-based solutions, the page below were gleaned from large customer implementations or common patterns: https://docs.microsoft.com/en-us/azure/architecture/browse/ Here is a post with some examples of IaC (Terraform, Ansible, etc...). https://old.reddit.com/r/devops/comments/smfr2e/do_any_compa... All of darklang's infra is source available, feel free to read it. This [1] is a good entry point for the infra configuration and setup. This doesn’t really answer your question, but gleaning it from job descriptions is one way I do it. If I’m curious how a company did something, searching for their job descriptions can turn up interesting stuff like what languages and frameworks they use, and often from there you can infer what their architecture might look like. Discord did a blog post a while back about their architecture https://discord.com/blog/how-discord-handles-two-and-half-mi... Check meetups.com and see if there are local DevOps or other technical meetup groups where people are demoing. its http://meetup.com/ I think Etsy has a "Code as Craft" blog with lots of interesting reads. It's not been as active the last 2 years but has been re-launched with more posts the last few months. https://techengineering.io/ aggregates the technical blog posts from various tech companies and they can be sorted and filtered based on reading time and the architecture you are interested in. The doom source code was released. If you would like a guided tour, maybe look at https://fabiensanglard.net/gebbdoom/ Unreal is source available too, if game engines are of interest to you The book "System Design Interview" by Alex Su and Sahn Lam is a good place to get digestible examples. It walks you through step-by-step how you might solve various systems problems, introducing the pieces you need. Each problem fits your 20-60 min request perfectly. https://github.com/donnemartin/system-design-primer It has link to many other articles and tech blog, besides having a lot of great info on system design and arch AWS Summit is approaching. I usually find other teams, even nominal competitors, or hulking behemoths of industry, to be quite proud of what they've built, and generous with their battle tested knowledge. All you have to do is reach out and ask ;) Great plan. Additionally, there is a “This is my architecture” series on YouTube where AWS interviews users. https://youtube.com/playlist?list=PLhr1KZpdzukdeX8mQ2qO73bg6... https://aws.amazon.com/architecture/this-is-my-architecture/ Be careful with things like this - it is ultimately an AWS marketing channel, so they’re not going to bring on people who say “we tried running everything on Lambda and it turned out to be a deployment nightmare”. The very best way to do this is find a tech meet-up around the sort of thing you do, and then go for the after event drinks. Get to know people, chat with them, and find out all the many ways architectures can shoot you in the foot. Here's a video of the technical (and product) evolution of Mailinator.com - and how each influenced the other. Software engineering blogs https://github.com/kilimchoi/engineering-blogs BuiltWith - doesn’t give you a whole picture, but it does share a lot of information. You can use stackshare.io to get an idea of different tech stacks companies use. This might shed some light on their architectures I'm a big fan of Sean Parent from Adobe, he has a lot of good material on YT; somewhat C++ specific though. I know it's not a 20-60 min example, but I find reading open source repos very informing:. I'm the cofounder of Budibase, and I like to jump on a call with new contributors and take them through the high-level arch and repo:
https://github.com/Budibase/budibase Enterprise Fizz Buzz: https://github.com/EnterpriseQualityCoding/FizzBuzzEnterpris... LAPSU$ leaks