Over the last year, it’s become abundantly clear that automating coding is not the same as automating engineering. Engineering is not solved, and it’s not even close.
On the one hand, models have made rapid progress. Agents just solved an entire Millenium Problem. In February, Dario said that engineers would be out of a job in 6-12 months. It’s September now.
On the other hand, I honestly feel that AI coding capabilities are roughly the same now as they were a year ago.
Certainly, infrastructure components like agent harnesses have seen large improvements, but I can’t shake the feeling that reports of the death of engineering are greatly exaggerated. I know many others share this sentiment.
Quite frankly, I want to believe that this feeling is true, that we will live in a future where craft and thoughtfulness still matter. And I want to live in a future where AI shares these values with me.
But is this only a feeling from hopelessly naive, luddite engineers, or does it poke at a deeper truth? To answer this question, I posit the following two theses:
- Proposition 1: software engineering is a game of balancing tradeoffs; there are rarely perfect solutions.
- Proposition 2: because there are tradeoffs in engineering, some entity must take responsibility for them. For the foreseeable future, that entity must be human and not an AI system.
Proposition 1: there are (almost) no perfect solutions in software
To set up a straw-man argument, you might ask: shouldn’t I just verify that all my code is correct via strict, rigorous specifications? This approach clearly works well in math.
To understand why this doesn’t apply cleanly to engineering, let’s set up a rough formalization of an engineer’s job:
Engineers write programs made up of code. The behaviors of these programs are parameterized by the vast space of possible inputs they accept.11 In pseudocode:
result := program(input = { customer_actions, etc .. } )
Engineers optimize programs so customers are satisfied with the result they return. One way to get some guarantees about this process is to define an objective function – oracle(result) – that outlines what the result should be, and then use techniques like testing or formal methods to verify that the program satisfies the oracle.
This presents our first problem: the oracle objective function is difficult to specify in a broad swath of applications. How do you write a mathematical spec for a “great user experience”? Even technical concepts like “availability” in distributed systems are surprisingly tricky to pin down.22 The best approach in many cases is to make our best guess and deal with the uncertainty.
Second, even given an oracle, there is a huge space of possible (program, input) state pairs. Any kind of verification requires these to be intimately connected to the oracle function; changes or uncertainty in one variable deeply affects the others.
Third, there is a time element involved in both of these things: the answers to “is this approach correct” are often not obvious until a quarter, year, or even 5 years later.
Taken together, it’s pretty clear that these problems are hard, and there’s often not a single right answer. Particular systems can still be verified, but people with domain-specific knowledge need to choose the tradeoffs, assumptions, and architectures under which this is possible.
There are innumerable stories of this process in practice across the industry: for example, a Rust rewrite at Discord in 2016 would have been foolhardy, but in 2020 it was wise. Making the right tradeoffs at the right time is at the heart of great software engineering.33
Proposition 2: humans must own the risk of engineering
If one can’t rely on perfect solutions in engineering, it means that businesses need to establish trust in the tradeoffs that’ve been made. To make this concrete, imagine that you must choose between two potential approaches to solve a problem that you’re facing.
Because writing software is an imperfect science, there’s some probability that each approach is wrong. In order to own this risk, you – as the stakeholder – need to understand the tradeoffs involved in each.
AI models don’t naturally fill this understanding gap because of illegible decision-making, in part because they are not (fully) aligned. On the other hand, with my human colleagues I have shared incentives and a workable theory of mind. LLMs give me neither affordance, and it’s hard to tell in advance how they’ll perform on a particular task.
On the other hand, in these same situations which demand trust, humans are in many cases stronger than models today. For example:
- Checking nonverifiable work: does this user experience “feel good”?
- Making verification easier by iterating on software architecture.44
Barring a massive jump in intelligence in these incredibly difficult tasks – which, in my opinion, would have far-reaching consequences beyond just engineering – it’s hard to see these things changing for the foreseeable future.55
For humans to plug in and contribute, though, they need to understand the system they work with (blog). This process doesn’t need to be painful, we just need better tools.
For example, imagine that a friend asks you a question, but you have to find the answer by leafing through every single book in a library, one at a time. This would suck! Thankfully, we have tools to make our lives easier: we have search engines, book indices, and the Dewey Decimal system.
Similarly, there are going to be software systems that will make it much easier to understand code, so that humans can provide valuable specification and architectural feedback.
Computers can and should be like bicycles for the mind.
Preparing for the future
In my opinion, the software teams that are going to win in the future are going to use AI extensively to accelerate their software development lifecycle (SDLC). However, humans will ultimately be responsible for these software systems, just like they were in 1979: “a computer can never be held accountable, therefore a computer must never make a management decision.”
For a glimpse of what the future might look like, we can look to the past. In 2019, the Golang project had too much contributor talent available, and not enough maintainer time (sound familiar?)
The project explored a number of ways to address this (Russ Cox’s blog):
- Sending maintainers design documents which focused on high level architecture and behavioral specifications instead of low-level PRs
- Giving maintainers clarity about the set of possible design decisions and their tradeoffs (scaling with decision documents)
- Attaching prototypes to proposals for review
- Explaining complex changes with multimedia - including video!
I think in the future, we can think of humans as the “maintainers” collaborating on combinations of specifications and software architecture, with AI agents as the “contributors” filling in the gaps. Taking all of this into consideration, I’m optimistic about the future. When man and machine work together, the whole might just be greater than the sum of its parts.
-
for example, potential customer actions, but also runtime infrastructure like networking, scheduling, etc. ↩
-
For example, “a request will take < n seconds to complete” is incredibly difficult on any kind of realistic hardware, which has all sorts of caches, speculative execution, etc. See also: https://dl.acm.org/doi/10.1145/379486.379512 ↩
-
you can think of these decisions as narrowing the possible (
program,customer_input) pairs, enabling verification itself. ↩ -
I have a feeling that software engineering is “AGI-complete” somehow. If, for all engineering tasks, models can beat a human at specifying the system and writing a satisfying proof for it (e.g. via architecture), I’m not sure how that kind of intelligence doesn't generalize. The ability to decompose an arbitrarily complex problem with uncertain rewards in the future feels diametrically opposed to the limits of models today (stunted by RL in a sense, a la Karpathy’s “sucking signal through a straw”) ↩
-
or require fewer people to do it (see hardware verification, where ratios of humans verifying vs. implementing are 3-5:1; https://blogs.sw.siemens.com/verificationhorizons/2022/12/12/part-8-the-2022-wilson-research-group-functional-verification-study/) ↩