Common misunderstandings about large software companies
philipotoole.comMany of the author's rebuttals hinge on the assumption that everyone in an organisation is acting in its interest first - and not their own, often conflicting, self-interest. As such, they are not particularly convincing.
Large organisations absolutely do, as a function of their scale, produce pockets where slackers and incompetents can hide. They'll surround themselves with a web of process, pointless meetings, and substance-free buzzword-heavy documentation/presentations to disguise this fact. Others may become ensnared in this web, and will rightly express the criticisms that the author is attempting to debunk.
I don't think I've ever worked at a company where slacking off was the problem. The vast majority of people want to do good work.
What I _have_ seen is several companies afflicted by this really strange characteristic of the software development industry: We appear to be the only industry on the planet where it is common to pick leaders (executives) that know nothing about the product or how it's made.
You can't run a bridge building company without knowing how to build a bridge. You can't run a law firm without knowing law.
You don't need to know all the nitty gritty - big picture is important - but understanding the product _in depth_ is a requirement in any business.
Are you in a completely different world than me? Because even the CEO of Boeing is not an engineer. Larry Ellison The CEO of the biggest bank in my country holds a masters degree in business economics, but nothing related to finance, econometrics or risk management. The CEO of US steel is an accountant. Don't even get me started on the (non)education of some politicians.
Understanding the product is often important, but equally often it is something you can delegate to others. It's only the younglings that think intimate knowledge of the product is the hallmark of a great leader, because that is the only thing they themselves bring to the table.
I agree fully with your comment, but I wish to point out that Larry Ellison's was a programmer at the time that the company that became Oracle was founded by him and his co-founders.
I get the point of his comment but it’s just nonsense… plenty of good CEOs aren’t SMEs in the field and plenty of bad ones are. the CEO of Boeing is absolutely an engineer - and so was the CEO during most of the years people consider the worst in Boeing’s quality history with the 737 Max (Muilenburg).
> Are you in a completely different world than me? Because even the CEO of Boeing is not an engineer. Larry Ellison The CEO of the biggest bank in my country holds a masters degree in business economics, but nothing related to finance, econometrics or risk management. The CEO of US steel is an accountant.
Specifically the CEO is more like the figurehead of the company; this role is to present and "sell the value" of the company to investors, important customers and partners. So often it is not too worrysome if the CEO has a different background; sometimes this can even make sense.
What should worry one much more is if the leadership layers below come from a very different background than what the company's industry is.
> Are you in a completely different world than me? Because even the CEO of Boeing is not an engineer.
Which is widely recognised as the root cause of that once venerable company's slow but inexorable downfall.
It's not always consciously slacking off. For example when I was at Google most of the team was simply incompetent. They thought they were smart (PhDs!) and working hard. But they refused to work together. They estimated tasks in weeks and months and at the end of my time there after I'd done very little due to obstruction by other teams I was praised for my high productivity.
I never saw anyone just sitting around or really slacking. But they couldn't execute anything. It was depressing.
> The vast majority of people want to do good work.
The vast majority of people have convinced themselves they’re doing good work. Then reject any suggestion they could do it better (including me).
Knowing how your product works does not actually solve the problem of running a large company.
No, but how are you going to run a sausage factory when you haven’t the slightest how the sausage is made?
Hire someone who does?
Indeed. Most people who seek employment at big tech do so because of the money, not because of the mission.
Spot on. There are disproportionately few people who care for the product. Most people care for making that fat promotion package so they can move up. Both eng and non eng.
I read it as saying that even if you solved the incentive misalignments, you would still have very similar annoying symptoms to what people complain about today. So you have to be careful in looking at any particular annoyance to disentangle which aspects of it are inherent complexity to a large company and which are BS. But I already believed that, so I may be steelmanning the article too much.
I think this is akin to x% of the worker ants doing all the work. Once you get to a big enough scale and have to delegate I'm sure every company hits this.
I just wish we didn't have to rely on hiring 100 on paper workers for 5 excellent people committed to the company...
> “There are too many meetings” At very large software companies, programming ability, technical expertise, and raw resources are not the limiting factors. Coordination is.
In my opinion there exist much more efficient ways for coordination: for example, write down some really good documentation and explanations that are then read by the other stakeholders, so that these, at the end, also have a very deep knowledge about the topic.
Nearly all employees have studied at a university, so the people are very used to writing texts (papers, seminar papers, lecture notes, thesis, ...).
In my experience the reason for too many meetings is rather that many managers love meetings.
--
> “There is too much process and bureaucracy” [...] At a very large software company, the software matters. It may be relied on by millions of people. It may underpin businesses, infrastructure, or daily life. It may not be particularly glamorous software but it has to work. It has to keep working. Failure is not charming, and recovery is not always cheap. [...] Process exists to manage risk, correctness, and scale. Calling it “too much process” without acknowledging the stakes involved is like criticizing a bridge for having too many safety checks because you once built a treehouse with a hammer and some nails.
This is one reason. Other common reasons for so much process and bureaucracy are
- Many managers love processes, because they can "hide" their failures behind processes, and introducing new processes and bureaucracy lets the manager pretend that he is doing something to solve the problems that plague the department.
- Many processes and bureaucracy are simply demanded by the legislature when you work in some heavily regulated industry. These legal demands often don't make sense.
> Nearly all employees have studied at a university, so the people are very used to writing texts (papers, seminar papers, lecture notes, thesis, ...).
I wish. Most people I've known in universities seem to read and write the absolute minimum to get by.
But I tend to agree that writing is preferable to meetings in most cases. I want to try out a policy that all meetings of more than two people must produce a written artifact, or clarifying edits to an existing document, that explains whatever ambiguity required a meeting to clear up. But you also need people to read. People don't read.
Insanely easy to game
What, the writing requirement? Yeah, you couldn't expect it to be airtight, more a nudge in the right direction.
Just because written communication works well for you doesn't mean that it works for others nor that it's the best way to communicate about everything. There's a place and time for both. For example with documentation, it's nice to have accurate and well thought out docs so you can search and read through it, but oftentimes it's faster if your teammate just tells you what bit you need. Meetings are the same way. We've all been in meetings that could've been an email, but that doesn't mean every meeting can be an email.
> write down some really good documentation and explanations that are then read by the other stakeholders, so that these, at the end, also have a very deep knowledge about the topic.
I wish. The reality at the moment is someone gets an LLM to do it and the other person uses an LLM to read it.
Meetings:
People who can write clear, unambiguous, accurate technical documentation are relatively rare.
And a meeting is in fact sometimes the best way of coordination. We have a question. We have five different people with relevant input into the question. Even if all five can write well (and will do so in a timely manner), we still need to reach a consensus on what the right answer is, and to make sure that everyone's issues are heard, and that they feel that they have been heard. A meeting often does that better than shooting emails back and forth among the five people.
Processes:
Processes are often added because something went wrong (often expensively wrong), and a process was created to make sure that it won't go wrong again. But what they miss is that the process also has a cost - a dollar cost, and a time cost.
Worse, there's a limit to how many processes most people can remember. You can create a process, that's the 47th process that your people have to remember, and when they don't keep it because they don't remember it, you can blame them. So here I kind of agree with you.
> Many processes and bureaucracy are simply demanded by the legislature when you work in some heavily regulated industry. These legal demands often don't make sense.
Maybe they don't. Ignore legal requirements at your own peril, though - they can have some pretty nasty teeth.
Somehow large open source projects have figured this out.
I've spent 20+ years in the industry seeking to understand first, and my conclusions have pretty consistently been that the systems are broken and inefficient as people already talk about. The caveat I'd add is that the critics themselves would also be just as much as a crapshoot.
Honestly it sounds to me like the author doesn't truly understand the inherent conflicts of interest at a large company. For example a really common anti-pattern is "Nobody knows X thing is a problem my team manages in a problem (e.g. our app eats battery usage), but if I draw attention to it they'll want to measure it forever. So do not make it a focus."
In short pretty much never does any employees/manager's/executives interest align with the company.
I liked this post. I may have some minor qualms (e.g. while I think execs should be proxies for the customer, they have many other competing motivations that can push customer needs way down), but I especially liked the closing section:
> Understanding before criticizing
> Large software companies have real problems. Some are structural. Some are cultural. Many are self-inflicted. But many of the behaviors people complain about are not pathologies – they are consequences.
> If you want to criticize how large organizations operate, it helps to first understand why they operate that way. Without that understanding, the criticism may feel sharp, but it will not be useful.
I see that kind of "criticizing before understanding" all the time on HN, and while that's probably just inherent to an open forum, commenters who do that should realize it makes them come across as "less than insightful", to put it generously. Like I see tons of comments often about how managers only get to their position through obsequious political plots. And sure, that may exist in some orgs. But you can always tell when folks have never even considered the competing forces that act on managers (i.e not just the folks they directly manage, but requirements coming from higher ups, and peer teams, and somehow being responsible for a ton when you actually have few direct levers to push) and solely view things through the lens of someone being managed.
Equating process to risk aversion is a mistake I hadn't seen before. It's true that moving slow makes it less likely to make something worse per unit time, but it also makes it less likely that anything will get better. It means that you don't try many things, and can't get important things done quickly.
You can ensure quality by making features opt-in, having a beta program available to risk tolerant users, adding QA resources, having representative users in captivity (employed at the company).
There is no law that says that you must move slow or do less in order to be low risk, you can also do a lot, move fast, and only let the best out.
Coordination is almost free in a ten-person startup. It is still relatively easy in a forty-person company.
Someone who says there are too many meetings is probably actually saying they are having bad meetings. If they got value from those meetings, they wouldn't be complaining. So there is likely still a problem to address.
Also, as a somewhat trivial side note, an instinctive reaction to not getting the clarity you need from a meeting is to ask for another meeting. So even if the optimal level of meetings is annoyingly high, bad meetings will probably push the level of (bad) meetings even higher. So you'll still actually have "too many" meetings.
> bad meetings will probably push the level of (bad) meetings even higher
bad meetings beget bad meetings
> If they got value from those meetings, they wouldn't be complaining.
This part actually felt quite relevant. Several years in the govt, and there was definitely a difference. Many meetings that felt inane, or meaningless to even attend, where you constantly questioned why you're even there, or bothered to show. Much phone swiping, and social media browsing. Often 10x+ attendance to people that ever participated. I often felt weird even asking anything, cause nobody was participating, and it felt wrong to even try to understand the endless charts on-screen.
However, a rare few that honestly felt quite worthwhile. We arrived, discussed what needed to said, and left with a better comprehension of the situation and the tasks necessary.
This important devil's advocate perspective reminds me of Chesterton's Fence [0].
I used to run a dev shop and had the opportunity to work with companies of all shapes and sizes. The startups often discovered Chesterton's Fence by declaring they didn't need this or that (meetings, accountability measures, etc), only to learn the hard way why they existed.
And meetings, beauracracy, et al are rightfully criticized for being inefficient and fostering mediocrity. But I think I'd agree with the author that it's glib to say meetings are dumb, no need for hierarchy without understanding their purpose
[0] https://en.wikipedia.org/wiki/G._K._Chesterton#Chesterton's_...
As a technology company scales up, making great software becomes one of a hundred things the company needs to do right in order to survive and grow. Doesn’t mean it isn’t absolutely essential, but so is having a strong GTM machine, finance competency, operational rigor, HR, and a long list of other essential functions.
It’s only the tech industry where the voice and ego of small companies hold outsized share of voice and love to claim the contrary.
> It’s only the tech industry where the voice and ego of small companies hold outsized share of voice and love to claim the contrary.
I'd be a little bit careful with this claim:
The fact that small companies can have such opinionated opinions without going bust is to me a sign that in particular for software development (but I don't claim that this is transferable to other industries) small teams/companies do have an efficiency advantage.
Many hypotheses can be formulated why this might be the case, like
- software industry is less regulated
- writing good software as the company's product requires a lot less collaboration between many stakeholders than what is necessary for producing other types of sellable products
- in software, "having a smart, though opinionated idea" is of a much bigger advantage (also for the company) than in other, more established industries
- ...
The author talks about how software creation processes at large organizations are an artifact of how large organizations operate, but in all 3 of the 3 cases, I would ask “why does the large organization need to be that way?”
Most obviously, why do executives need to be the proxy to customers? Why can’t development teams simply talk to real customers? This isn’t just an abstract idea in agile, it grew out of actual Japanese product development practices practiced at large organizations: Toyota, Canon, and others, and documented in “The New, New Product Development Game” HBR review article that was so influential to early agile.
The point that in large organizations, most of the work is coordination, again demands the question, why? It’s been understood since at least World War I by some military planners (with organizations far larger than Google) that coordinating dependencies was far more complex than reducing or minimizing them. Goldratt wrote about it when designing a project management system for Theory of Contraints (indeed, you could argue this is a fundamental learning of ToC). And one of my favorite software conference talks of all time is Mary Poppendeck’s excellent “Tyranny of the Plan,” where she notes that as computer systems have been used in planning, we seem to have become more confident but no more competent in coordinating, rather than focusing on flow, in large-scale projects.
Finally, on the importance of software created at large organizations, I agree, something that will have millions of users on day one has a greater responsibility, but that doesn’t mean that loads of bureaucracy and checking are the pathway to quality software. First of all, does anyone believe that highly scrutinized and bureaucratic functions are general high quality services? The often provide access to even the most extreme edge cases, but they do so by reducing the quality of service to everyone else. Anyone who’s ever filled out their own tax forms in the United States knows that it covers every base of possibly income, but 80% of people really only need to be concerned with 2-3 common forms, and 99% could simply be asked about 10 forms or so. Instead we have to answer questions for “directors of foreign corporations who also happen to be Us citizens,” instead of just requiring those people to fill out an additional form. And, of course, to (probably badly mis-)quote Deming, “you can’t check quality into a product.”
I would turn it around in the author: yes, the software practices operate this way in large orgs because large orgs are structured differently—but why do large orgs need to be structured that way? Is it inherent when absentee owners with low domain context (shareholders) pass ownership over to a manager? Is it because hierarchies insulate good but not great managers from genuine value creation as long as they play politics well? Is it because these are first order ways to understand complexity, and again, low-context absentee owners aren’t going to do the work to understand the more complex dynamics at play?
Blog post author here, thanks for your thoughtful comment. You raise some interesting things to think about.
>First of all, does anyone believe that highly scrutinized and bureaucratic functions are general high quality services?
This is the only part of your response that doesn't quite sit right with me. There could be many "highly scrutinized and bureaucratic functions" out there that are working very well, you just don't notice because they work so well. There could be a selection-effect here.
Quality is a big deal for me[1]. But I think you're defining "quality" too narrowly in this context. "Quality" could also mean "allows everyone, at scale, reliably, to do what they need to do." The US Tax Filing system (and its associated software) meets that goal.
[1] https://philipotoole.com/always-thinking-of-the-next-guy/
Oftentimes, broadly accessible services are lower quality than more personalized ones, to the consumer.
As an example in US government bureaucracy, government software teams digitizing forms at one point weren’t allowed to utilize features like autofill or automatically filling fields based on previous answers because it would relatively disadvantage users using paper forms.
Government capabilities do need to serve everyone, and from the perspective of the whole society that is beneficial, but they are often are of low quality to the individual consuming them for this very reason.
Let’s exclude taxes, because obviously many people would hate them under any circumstances. Does the government do a good job providing the other services people interact with regularly? Do people love their visits ti the DMV? Are they satisfied with their interactions with the police? Heck, in my town, just renewing a dog license is a pain.
> The US Tax Filing system (and its associated software) meets that goal.
I disagree with the argument that the US Tax Filing system meets the goal of:
> "allows everyone, at scale, reliably, to do what they need to do."
It may do so for easy / common cases of W2 salaried employees but step a little outside of the norm (foreign sourced income, tax treaties etc.) and software gives up and shows you a PDF of relevant forms and requires you to become an expert in tax code and to keep your own multi year running calculation of carryovers and things to proceed. I'm glossing over all of the detail about how complex this really is but wouldn't expect the average, even very intelligent person to succeed in filing a correct return without a professional's help.
Meetings aren’t a scale problem, they’re a clarity problem dressed up as collaboration.
When execs dominate decisions, it’s usually because the org failed to make customer truth unavoidable.
Most process exists to protect careers, not outcomes.
Understanding why something exists doesn’t make it justified or effective.
None of this seems particularly revelatory to me. It comes across more as just an argument against letting software companies get large.
I don't get it. Article titled "Common misunderstanding". OK. “There are too many meetings”. How is this a misunderstanding? "...bunch of excuses confirming there are too many meetings...". Yyy... so it's not a misunderstanding?
> At very large software companies, programming ability, technical expertise, and raw resources are not the limiting factors. Coordination is.
If coordination is your limiting factor I'd argue that it shouldn't be, and you're not investing enough in removing it as a factor. Companies can use various tools to do this, for example:
* Defining directly responsible individuals / single-threaded leaders so that every choice doesn't involve massive coordination
* Putting people who work together in the office sitting next to each other most days of the week
* Or, for remote work, having a strong culture of async communication that is visible to the broader group by default, for example with Slack
> * Defining directly responsible individuals / single-threaded leaders so that every choice doesn't involve massive coordination
Obviously doing this makes sense when possible, but in cases where you see it, it's usually due to a fair amount of work to have made this possible.
To give a simple example, I work on a class of problems that require a VP level approval for new instances of $thing. I've gotten this down to a pretty straightforward process where I can work with folks who are proposing a new instance, get things working, and then the VP usually is happy to stamp the work we've already done. Though in some cases they aren't, and they have (good, probing) questions or changes they'd like us to make. But that's only possible because I have a longstanding relationship with the set of folks who ultimately approve this kind of thing, I have worked with them on a process that they like, and I have the experience to shepherd things effectively, and their trust and buy-in. And I'd argue that my case is pretty simple, because while there are a handful of responsible people that I could go to, there's rarely any individual concerned.
Consider a different, relatively common case, of a client and a server that are owned by different teams. The client wants a feature in the server, and perhaps is willing to do some development and loan headcount to implement the feature. Who is the single responsible party here?
It should probably be either the manager of the client server team or the mutual manager of both if the teams are fairly close in the wider org, but then you have multiple indirect layers between the people doing the work (client team members) and the accountable person (server team manager), and that's the state you get to after you've done a bunch of coordination work and gotten everyone to agree to that division of labor and accountability.
One nitpicky detail is that the executives may be a rep for the customer/consumer, but are also very much reps for the shareholders and that’s a pretty big distinction
I prefer Parkinson's take on the matter. Parkinson's law has a lot more to do with why companies grow so large. Why WhatsApp can can't write meaningful software with fewer than 10 people
I did not expect to see big-company apologia on Hacker News.
https://en.wikipedia.org/wiki/Apologia
The thing this (very good) post doesn't mention is that big companies select for blub languages because that's where the most low-cost labor is, in that you can hire multiple Java developers for the cost of one Haskell developer, even if Haskell might be objectively a better choice for the project.
I don't think this is a charitable interpretation. As a business, you need to be able to backfill positions or hire more when the need arises. If you use a language that's very commonly used, it's a lot easier to hire. There isn't anything sinister to that, it's simply reasonable.
There was a post, I think on the Uber engineering blog, that stuck with me. It essentially boiled down to: it's easier to change the tech stack than the hiring pool, and talked about deliberately setting something up that was technically less optimal but easier to hire for
Corollary: it's perhaps easier to throw money at fancier hardware to improve performance, than the alternatives
> I did not expect to see big-company apologia on Hacker News.
In the comments I see little apologia. The article rather brings up some points which are contrarian to the common view on HN, and the people on HN discuss whether these points have some truth in them, or the author missed some important considerations, or whether the author is wrong.