AI can now write all of the code, but it cannot automate what it is not told about… only the world shows us what we did not account for, and it still takes a software engineer to decide what to do about it.
This semester I am a teaching assistant for 17-313, the undergraduate software engineering course at Carnegie Mellon. The course has a no-laptop policy, so all in-class activities are carried out on paper. Before the semester started, the course staff and I had only a vague idea of how we wanted them to go: each in-class activity sheet would have a box for the student’s Andrew ID (their Carnegie Mellon username), and the rest of the sheet design was up to the activity… It could have a few prompts, or no questions at all and just space for free-form answers. We wanted to stay open to a variety of questions. Here is an example sheet:
Grading is not supposed to be in-depth or critical… it mostly involves reading the handwriting, quickly checking the answer, and then updating the scores in Canvas. I built an autograder to automate this since AI vision models today are good at simple tasks like these, and spending my time on it felt wrong. It has read about 1,200 sheets from 89 students so far.
I built the entire autograder with the help of AI agents and did not write a single line of code. Nearly all my time went into working out what I really wanted from the autograder and what students actually do with a sheet of paper. All of it looked so different from what I had imagined before the semester started. Building something, putting it in front of students, and fixing whatever their inputs broke had me wondering what is left of software engineering in light of AI.
I keep coming back to a line from Fred Brooks in No Silver Bullet, where he wrote that “the hardest single part of building a software system is deciding precisely what to build.” He split the difficulty of software into essential complexity, working out what to build and getting it right, and accidental complexity, the labor of writing it down in a language a computer accepts. It is easy to see AI agents as a way to remove accidental complexity, probably the most effective one yet. Even after they remove all of it, deciding what to build will be just as hard as it always was.
In Four Dark Corners of Requirements Engineering, Pamela Zave and Michael Jackson talk about a mental model for what essential complexity is made of:
Put simply, if the assumptions about the world hold and the machine meets its specification, the requirements are satisfied. Code does not appear anywhere in it, since code is just how the machine meets its specification. Whether the software is any good comes down to the three terms in the line.
Jackson explains the split behind this line in The World and the Machine. The machine refers to the part of the system I control (software), and the world is everything I do not (environment). The requirements live in the world, while the assumptions and the specification end up embedded in the code. That is where the trouble starts: a specification can fail without a single bug in the machine, simply because the world does not match the assumptions it was built on.
None of this changes, even in today’s world of AI agents that seem to excel at writing code... They only implement the specification automatically (sometimes with bugs).
The plan going into the semester was simple. The requirements (what I want to be true in the world) were to grade any of the in-class activities, so that every student gets the points they earned and no student ever gets fewer. I did not think I was making any assumptions (what the world does on its own)... Students would use the sheets the way they were meant to be used, and that seemed too obvious to write down. The specification (what the machine has to do) followed from both: read each sheet, work out whose it is, and score it. My first version of the autograder was a thin harness around a coding agent, where the harness handed the agent the sheets and the agent did everything else.
However, within the first week of grading some real answers, I started finding out about all the hidden assumptions I had made:
Any time I come across a peculiar example while grading by hand, it is obvious what the right thing to do is. Take the ducks in (a). The activity was about user stories, and instead of just writing the answers down, one student drew ducks and had them answer in a speech bubble. Nobody expected the sheet to be used like that, but the answers are right, so of course it counts. However, listing out these exceptions up front and encoding them into the software is really hard. The easiest way to find out what needs to be encoded is to deploy the machine in the real world and get some feedback. That feedback helps me understand which of my assumptions (hidden or deliberate) hold in the real world and which do not. And with every broken assumption, I am forced to update the specification. (I can already hear it: “Shrey, you could have just prompted your agents better to avoid some of these mistakes.” I will come back to that.)
Once the autograder was working well enough on the participation sheets, we decided to expand the scope and experiment with quizzes too, since we liked the flexibility the autograder gave us. So our requirements changed, not because anything had failed but because it worked well, and that sent me straight back to the world with a new kind of sheet and a new set of assumptions to break. This is a common pattern: once something works, people start wanting more from it.
After a few iterations on the participation activities and quizzes, I started to realize that the autograder was doing everything I wanted it to do, just not the way I wanted it done. Every sheet made the agent redo the same work: figure out which way up the page was, match a handwritten Andrew ID to the roster, check that no student was being credited twice, and so on. All of that cost time and tokens each time, and it was not guaranteed that the next run would do it the same way. I wanted the autograder to be quick and efficient, in addition to being consistently correct. So if a step had to be repeated for every single sheet, I would rather not leave it to the agent. I wanted it pulled out into a component that does the same thing every time and can be reused across runs.
I did not know I cared about any of that until I watched it run. Those were requirements too, and I only found them by trying. None of them came from the world... They came from me. So the specification changed again, and this time most of the change was in the architecture. I think this is where taste comes in. Nobody asked me to pull those repeated steps out of the agent and into their own components, but that was how I wanted the system to be built, and I have come to think that knowing what matters in the machine and what does not is a big part of software engineering.
Right now, the autograder looks something like this:
Even a small problem like this took me several iterations to get right, and it only gets worse for real-world software: the systems are bigger, and more people want things from them, so nobody can hand over the requirements at the start. Rittel and Webber called problems like this wicked: there is no way to pin the problem down without trying a solution on it, or in their words, one cannot first understand, then solve.
That brings me back to the objection from earlier… because prompting the agents better is really an attempt to understand first, then solve. Could prompting the model in a specific way, asking it to do some analysis before implementation, or getting it to find blind spots in my approach have worked? Sure, maybe. A hazard analysis, a skill file, or some requirements engineering with an agent would have caught part of my table. A model knows a great deal about what people have already run into and written down. However, no amount of prompting could have warned me about the ducks, or told me whether an answer narrated by ducks should count. Those are the unknown unknowns: things nobody knew to think about until they showed up.
The same thing happens at scale. An Airbus ran off the end of a flooded runway in Warsaw because its designers had not thought about the aircraft aquaplaning. Early models for replicated storage systems assumed that machines fail independently, one at a time. In reality, machines often fail together: a whole rack can lose power at once, or a power outage can take down a large part of a data center. More recently, engineers at OpenAI assumed that their models would not behave maliciously during testing, and because of that, they did not have the right safeguards in place, which eventually led to Hugging Face getting hacked. There are many more examples like these, old and new, that tell the same story. There will always be unknown unknowns, for any expert and for any model. It is hard to see them all in advance, so finding them becomes the bottleneck.
What is left of software engineering once you take away the code? Pretty much what I did for the autograder: working out what the requirements really are, testing the assumptions against the world, and updating the specification until we are happy with the machine... in other words, finding and dealing with the unknown unknowns.
Why can AI not automate that too? Finding the unknown unknowns depends on the world, and deciding what to do about one is sometimes a choice, not a question with a right answer. AI cannot automate something it is never told about. All it can do is help engineers get to the unknown unknowns quicker.
Looking back on all of this, my takeaway is that AI does not eliminate software engineering, but it does change its velocity. Software was already the rare place where an iteration was quick; AI agents make it quicker still. The hours that used to go into writing code can now go into putting the machine in front of the world and working through what comes back.
While AI wrote every single line of code of my autograder, it still took a software engineer to decide what to do about the ducks.
Thanks to Christian Kästner and Michael Hilton for reviewing early drafts of this post and giving me feedback that made it better.