Exponential Failure

In 1942, Robert Lusser joined the German aircraft maker Fieseler as one of the principal engineers on the Fi 103, the pulse-jet flying bomb better known as the V-1. Many early V-1s failed in testing. After the war Lusser joined the American rocket program in Huntsville, Alabama, and his name became attached to a piece of arithmetic that explains why complex machines fail so often, and that has outlived the weapon by decades.¹

The everyday intuition about machines is that a chain is only as strong as its weakest link: find the least reliable part, improve it, and the whole improves to match. Lusser’s arithmetic said the chain is weaker than that. When a machine works only if every one of its parts works, its reliability is the product of theirs, and multiplying numbers smaller than one always gives something smaller still. A machine of a hundred parts, each of which works 99 percent of the time, works about 37 percent of the time. No single part is anywhere near that bad. The machine is.

Reliability engineers now call this Lusser’s law, or the product law. A colleague of Lusser’s, the mathematician Erich Pieruschka, noticed that when every part is equally reliable, the law collapses into a single expression: pⁿ, the reliability of one part raised to the power of the number of parts.² The law sits in every reliability textbook. It shaped how missiles, aircraft and power plants are designed. It is seldom applied to software, and that is about to matter a great deal. AI coding agents are multiplying how much software gets written, and almost no one is asking what Lusser’s arithmetic says about all those new parts.

A software project is a machine of exactly this kind, though its parts are harder to see than bolts and valves. Consider something modest: a new checkout page for an online store, built the way software is increasingly built, by machines from end to end. One agent turns a request into a specification. Another turns the specification into a design. A third writes the code, a fourth writes the tests and a fifth reviews the result.

Writing the code takes minutes, and it is the least of what has to go right. The libraries the agent chose to pull in have to work, and keep working. The tests have to test the right things, not merely pass. The payment provider’s interface has to behave the way its documentation says it does. A change to the database has to run cleanly against years of accumulated records. The specification has to capture what the request actually meant, and every handoff from one agent to the next has to carry that meaning intact.

Each of these is a part, in Lusser’s sense, and the project works only if every one of them does. Even a project this modest has fifty such parts.

Now suppose the agents are good, the handoffs between them are clean and we’re lucky, so that each part has a 95 percent chance of working. Suppose further that mistakes caught in time cost nothing: if an agent gets something wrong and it, or another agent, fixes the problem before anything is built on top, the part counts as working. Taken at face value, Lusser’s law puts the chance that all fifty work at 0.95⁵⁰, or about 8 percent. Roughly one such project in thirteen ships what it set out to build. (Experienced engineers will know that real projects are messier than the law assumes. Their objections, which we will come to, bend the arithmetic without breaking it.)

Accidents without culprits

At odds like these, failure is the normal outcome. Even in a project as modest as this one, with just fifty parts, a mistake has usually slipped past every agent. Nothing points to it. The code compiles. The tests pass. The reviewing agent approves. The project looks finished, and so it ships.

The mistake surfaces later, as a symptom rather than a cause: a feature that does not quite do what was asked, a number on a dashboard that nobody can explain, a fix that breaks something else. By then, more work has been built on top of the faulty part. The cleanup gets filed under technical debt, a phrase that implies a decision to borrow. There was no such decision. Anyone who goes looking for one comes back empty-handed. Every part performed about as well as it reasonably could. The failure lived in the multiplication.

The sociologist Charles Perrow reached a similar conclusion about industrial disasters in his 1984 book Normal Accidents. In systems that are complex and tightly coupled, he argued, serious failures are not aberrations traceable to a single culprit. They are a normal property of the system’s structure.³ Software projects are rarely dangerous in Perrow’s sense, but they share the structure, and the search for a culprit comes up just as empty.

What happens next is where organizations tend to make things worse. Faced with a project going off the rails for no nameable reason, they reach for remedies. They add agents. They add an orchestration layer or an approval gate. They put another model on review. They swap in faster, cheaper models to make up lost time. They tell an agent to throw out the module that keeps failing and start over.

Most of these remedies make the odds worse. Fred Brooks observed the pattern among people in The Mythical Man-Month in 1975: “Adding manpower to a late software project makes it later.”⁴ Adding agents does the same, and does it faster. Each new agent adds handoffs and assumptions, and every one of them is another part that has to work. It also produces parts of its own. An orchestration layer or an approval gate adds parts as well, and so does another reviewing model, which is one more part, with blind spots of its own. Faster, cheaper models, chosen to save time rather than because they suit the work, usually make every part less reliable, the machine equivalent of working tired. Throwing out the module that keeps failing and starting over is a gamble. Done with a clear diagnosis of why the module failed, a fresh start fixes the part that matters. Done in haste, it swaps known problems for unknown ones. Taken together, these remedies resemble an autoimmune reaction: the body’s defenses, mobilized in good faith, attacking the body.

Part of the reason these remedies persist is that each comes with a respectable story. Acting on instinct feels like leadership. Adding process feels like discipline. Adding agents feels like progress. Doing the arithmetic feels like homework, and it returns an answer nobody in the room wants to hear, so it rarely gets done.

The squeeze

One remedy deserves a closer look, because it sometimes works and because agents make it tempting: compressing the project’s schedule, on the theory that the machines can make up the difference. Squeeze a project and things get cut, and fewer parts means fewer chances to fail. All else equal, that improves the odds.

All else is seldom equal. Rushed work is worse work, even when machines do it: verification passes get cut, cheaper models stand in for better ones, test suites get trimmed to fit the deadline. Take a project of fifty parts, each with a 95 percent chance of working. Suppose a squeeze cuts ten of those parts, and the rush drops each remaining part’s chance of working to 90 percent. It sounds like a reasonable trade. It is not. Each remaining part now fails twice as often. The project has 20 percent less work to do and an 80 percent lower chance of success.

Bad as that sounds, the real picture is worse, because so far we have treated failures as free. They are not. Each one generates work: a bug to find, a fix to review, a retest nobody scheduled. Each of those is another thing that has to go right. So each failure adds parts, which lowers the odds, which produces more failures. Failure breeds failure.

There is a bright side, though a narrow one. A squeeze can hold a project’s odds steady, or even improve them, if the rush does little damage to the parts that remain. How little? In the squeeze above, cutting ten of the fifty parts pays off only if each remaining part’s chance of working falls by little more than one percentage point.

To be clear, the cut is never the problem; the rush is. In principle, scope can be reduced deliberately, with the remaining work given the checking it needs. In practice, the checking rarely happens, because of how it looks from the outside. An hour an agent spends producing new work shows up as progress: new features, new code, a longer list of things done. An hour it spends checking work it has already done shows up as nothing at all. On a dashboard, diligence looks exactly like idleness.

More parts, worse odds

Two numbers decide a project’s odds: how many parts it depends on, and how likely each part is to work. Coding agents change both. They certainly increase the number of parts. They may also make each part more likely to work. The question is whether they can do the second fast enough to keep up with the first.

For as long as software has existed, a project could have only as many parts as people could write. Machines now do the writing, and that limit is gone. An agent can produce parts as fast as anyone can ask for them, and people never run out of things to ask for.

Every one of those parts has to keep working, not just on the day it is written but every day it runs, every time something new is connected to it, and every time the vendor behind it ships a change.

Adding got cheap. Owning didn’t.

All those extra parts do more damage to a project’s odds than intuition suggests, because the damage grows exponentially. The numbers are worth spelling out. Fifty parts, each with a 95 percent chance of working, all work 8 percent of the time. If an agent doubles the parts, which it can do without breaking stride, the hundred parts all work 0.6 percent of the time. If it multiplies them tenfold, to five hundred, the chance that every one works falls below one in a hundred billion.

The obvious hope is that the models will improve enough to keep up. Unfortunately, they would have to improve a great deal, because the arithmetic sets a brutal pace. Every time the number of parts doubles, each part’s chance of failing has to be roughly cut in half just to keep the project’s odds from falling. Even if each part had a 99 percent chance of working, five hundred parts would all work just 0.7 percent of the time.

The good news is that the models are improving fast. By one widely watched measure, from the research group METR, the size of the software task an AI agent can handle has been doubling every few months.⁵ The bad news is that their progress has a side effect. Every time the models get smarter, faster or cheaper, people hand them more to do: bigger projects, more features, work that used to be done slowly and carefully by hand. Economists know the pattern as the Jevons paradox, after William Stanley Jevons’s observation in 1865 that more efficient steam engines had made Britain burn more coal, not less.⁶

Better models, then, cut both ways. Every improvement raises a project’s chance of success, and every improvement invites more parts, which drag that chance back down. The reliability a project needs from its models is not a fixed bar; it rises with every part the agents add. The models are victims of their own success. Human nature always wants more, and arithmetic punishes every addition.

None of this is hypothetical. It already shows up in the field. For two years running, Google’s DORA survey of thousands of software professionals has found that, on average, the more a team leans on AI, the less stable its software.⁷ The cause, its researchers say, is volume: more changes mean more breakage unless testing keeps pace.

Martin Fowler, whose book Refactoring taught a generation of programmers to clean up their code,⁸ described the same trade years before coding agents arrived: speed now, trouble later. A team that skimps on what he calls internal quality, the parts of the code users never see, can ship features faster for a while. The advantage doesn’t last. Developers find, he wrote, that poor-quality code “significantly slows them down within a few weeks.”⁹

A Carnegie Mellon study of more than eight hundred open-source projects has now watched machines repeat it. Projects that adopted the AI coding agent Cursor sped up for about two months. What they were left with was code that was measurably harder to work with, and it slowed them down.¹⁰

When people wrote the code, testing suffered for human reasons, and the chief one is that the best programmers are lazy. Larry Wall, the creator of the Perl programming language, called laziness, the urge to make a machine do the dull work, a programmer’s first great virtue.¹¹ Testing is dull work. It was the first thing to go.

Agents have the opposite problem. They don’t get bored, they don’t go home, and they never stop producing. The trouble with agents isn’t laziness. It’s tenacity. As the parts multiply, the checking has to multiply with them, and there are only two obvious ways to manage that. One is to check each part less, which is the squeeze again. The other is to find something else to do the checking, and nobody yet knows what that something is.

Finding it is the central problem in today’s debate over superintelligent machines slipping out of human control. Who, or what, checks the agents? How? And can the checking itself be trusted?

Reliability engineers have known part of the answer for decades, though the AI industry rarely mentions it. A backup protects only against failures it does not share with the thing it backs up. Engineers call the shared ones common-cause failures, and no amount of redundancy removes them.¹²

That is bad news for agents checking agents, because they share a great deal: similar training, similar habits, the same way of misreading an ambiguous request. Researchers at Cornell University have measured how much. They gave more than 350 language models the same multiple-choice questions and looked at the questions two models both got wrong. If their mistakes were random, the two would have picked the same wrong answer a third of the time. They picked the same one 60 percent of the time. The overlap was strongest among the largest and most accurate models, even ones built by different companies. Better models, in other words, share more of their mistakes, not fewer.¹³

The shared mistakes are also the hardest to spot. As a model improves, its obvious errors are the first to go, and the ones it still makes are the ones that look right.

An agent reviewing another agent’s code catches the mistakes it would not have made itself. The ones it would have made too, it misses. So would a third reviewer agent, or a tenth, with the same blind spots.

That is the bind. Agents multiply the parts, and the easiest way to check all those parts is with more agents, whose blind spots are the very ones they were brought in to cover. And if agents can’t be trusted to do the checking, what can?

The promised land

Parts that work 95 percent of the time sound good, and as we have seen, they are nowhere near good enough. Nearly perfect parts are a different story. At 99.9 percent per part, five hundred parts all work 61 percent of the time. At 99.99 percent, 95 percent of the time. The whole payoff is packed into the last fraction of a percentage point. Below it, a large project’s chance of success is indistinguishable from zero. Within it, the chance climbs almost to certainty. Machines able to produce parts that reliable could build projects of five hundred parts, and those projects would almost always work.

The more parts there are, the more each fraction of a percentage point is worth. For today’s agents, the implication is stark. Ten times the output means ten times the parts, and to keep a project’s odds where they were, each part then has to fail only about a tenth as often.

That is the requirement on paper, and in practice it comes down to this: a large project needs nearly perfect parts. Proving that it has them is remarkably hard. In 1993, Ricky Butler and George Finelli, two researchers at NASA, showed that testing alone cannot establish extreme reliability in software.¹⁴ The difficulty comes down to simple statistics. Suppose a part has run three hundred times without a single failure. It is tempting to call it flawless. But a part that fails once in every hundred runs will get through three hundred runs cleanly about one time in twenty, which is too often to rule out. So three hundred clean runs prove only that the part works about 99 percent of the time. Proving 99.9 percent takes three thousand clean runs. Proving 99.99 percent takes thirty thousand. Statisticians call this the rule of three.¹⁵ Writing a part costs the same whatever the goal. Confirming it costs ten times as much with each step toward perfection.

Second chances

Skeptics of this kind of arithmetic usually raise two objections. The first is that the parts of a real project do not fail independently of one another, as the multiplication assumes. The second is that a mistake need not be final. Both objections have merit. Neither rescues as much as it seems to.

Take the first. The skeptics are right that parts are not independent, but the dependence cuts both ways. When several parts share a cause of failure, such as a single misreading of what the software is for, the project has fewer separate chances to go wrong than it has parts, and the multiplication overstates the risk. But parts also have to work together, and the number of pairs of parts that can interfere with each other grows far faster than the number of parts. Counted that way, the project has more chances to go wrong than it has parts, and the multiplication understates the risk.

Which effect wins depends on what gets added, and agents rarely add more of the same. They add new libraries, new interfaces and new paths for data, each with its own ways to fail and its own ways of interfering with everything already there. Where a shared cause does dominate, it offers little comfort: a project whose parts all rest on one reading of the specification is only as reliable as that reading. Either way, the field evidence makes no assumption about whether parts fail independently. The surveys and studies that found software growing less stable and harder to work with as AI output rose measured what actually happened, and what happened is what the multiplication predicts.

Now the second. When we assumed that each part works 95 percent of the time, we already counted any mistake that was caught and fixed in time as working. The 5 percent that remains is the mistakes nobody noticed. The natural response is to notice more of them, by adding checks and having the agent redo whatever a check rejects.

On paper, this turns the arithmetic in the agent’s favor. A project needs every one of its parts to work, so each added part is another chance for something to go wrong. A faulty part gets through only if every check misses it, so each added check is another chance to stop it. Suppose every part goes through three checks, each of which catches nine out of ten faulty parts regardless of what the others caught, and suppose the agent eventually fixes whatever a check catches. These are generous assumptions, but they are worth following to see where they lead. A faulty part would then slip past all three checks only 1/10 × 1/10 × 1/10 = 1/1,000 of the time. A part that worked 95 percent of the time now works better than 99.99 percent of the time, well inside the promised land.

That reasoning fails because the checks are not independent. Some faults are simply harder to catch than others, and a fault that is hard for one check tends to be hard for the next check of the same kind. Researchers at City University in London worked out the mathematics in 2000 and applied it to an experiment in which people searched railway-signalling software for faults planted in it. Two people reading the same code were about a third more likely to miss the same fault than they would be if their misses were independent, and the researchers showed that treating checks of the same kind as independent always makes the odds look better than they are.¹⁶ When agents check agents, most of the checks are of the same kind: a model reading code, then another model reading the same code, often against tests that a model wrote.

Two kinds of fault are especially hard for checks like these. The first looks right: code that passes its tests and does the wrong thing. Today’s models are refined using feedback that rewards answers that look right, so when they are wrong, they tend to be wrong convincingly.¹⁷ The second comes from an ambiguous specification, which the agents writing the code and the tests can misread in the same way.

Nobody redoes a part that seems to work. It ships, and it is broken in production. Extra checks catch the easy faults cheaply. Against the hard ones, each new check does less than the last.

Those who notice that their checks are fooled by the same things tend to reach for diversity, for instance by having a model from another company review the first model’s work. Diversity can work, but only if it introduces different kinds of checks. In the London experiment on railway-signalling software, a person reading the code and a person testing it missed the same fault less often than they would have if their misses were independent, because each method was strong where the other was weak. Having models from two companies review the same code, however, is like having two people read it, not like having one person read it and another test it. As we saw earlier, models from different companies make many of the same mistakes, and the better the models, the more their mistakes overlap.

A bolder remedy is to have several agents each write the same part on their own, then run their code side by side and take the majority answer, so that a mistake in any one is outvoted by the others. Majority voting works only if their mistakes are independent. In 1985, two NASA researchers, David Eckhardt and Larry Lee, showed mathematically why they aren’t. Some inputs are harder to handle correctly than others, and even programmers working in complete isolation stumble on the same hard ones.¹⁸ Strange as it sounds, working independently does not make mistakes independent.

Experiments with people and with machines bear the theory out. In 1986, the computer scientists John Knight and Nancy Leveson had 27 students, some with years of professional experience, each write the same program from the same specification, then ran all 27 versions against a million test cases. The versions failed together far more often than independence predicts, and the programmers had made similar logical errors in the difficult parts of the problem.¹⁹²⁰ In June 2026, computer scientists at KTH Royal Institute of Technology in Stockholm and the Université de Montréal repeated the experiment with AI coding agents and the same specification. Five coding tools, running 23 models in three programming languages, produced 48 versions that passed an initial round of tests, and the researchers ran each against a million random inputs. The versions failed together nearly four times as often as independence predicts, and many of the shared failures came where the specification was hard or ambiguous.²¹ When the researchers formed every possible group of three versions and let each group settle the answer to each input by majority vote, the average number of failures fell by about two-thirds. Diversity is worth something. It is not independence.

The mistakes every model shares set a floor that no number of models can get under. Worse, adding models makes that floor harder to see, because agreement feels like proof. When several models approve the same code, the natural conclusion is that the code is right. It may only mean that they share a blind spot, and nothing in their agreement says which.

Second chances are real. A second try catches the mistakes that announce themselves, and a second model might catch mistakes that a first model misses. But second looks help only where they see differently from the first, and anyone or anything trained alike tends to see alike.

What benchmarks measure

If the arithmetic is this unforgiving, why do the models keep posting such impressive scores? Every few weeks, an AI company announces a new top result on some benchmark, and the numbers only seem to go up.

A benchmark measures how often a model gets a single task right. That is the closest thing anyone has to a reading of each part’s reliability, and it is worth having. But it is not the number that decides whether a project ships, which is the chance that every one of the project’s parts works at once. That depends on which parts the project has and how many, and no benchmark knows either.

Some benchmarks try to measure the whole rather than the part, by testing long tasks made of many steps, where every step has to go right, just as every part of a project does. They end up hiding the problem instead of solving it. Such a benchmark fixes the number of steps, say twenty, and reports how often all twenty succeed. A model that gets each step right 98 percent of the time scores a respectable 67 percent. On a task of a hundred steps, the same model would succeed about one time in eight. The benchmark’s designers chose the number of steps, and the number of steps is the whole problem.

The scores are also softer than they look. Economists call it Goodhart’s law: when a measure becomes a target, it stops being a good measure. Benchmark scores are now the main way AI companies market their models, which makes them targets. In 2025, a team of researchers examined Chatbot Arena, the most-watched public leaderboard for AI models, and found that a few large companies had quietly tested many private versions of their models there and published only the best. Meta alone tested 27 before releasing Llama 4.²² And once a benchmark’s questions are public, they can find their way into the data that later models learn from, so a high score can reflect memory rather than skill.²³

None of this makes benchmarks worthless. It makes them the wrong instrument for the one question an organization needs answered, which is whether its own project, with its own parts, will work. A public score reflects someone else’s tasks, someone else’s choice of how many steps, and someone else’s incentives. The only benchmark that can answer the question is one an organization runs on its own work, for its own eyes. The moment that benchmark becomes a marketing number, Goodhart’s law applies to it too.

The antidote

The news so far has been almost all bad. More agents add parts. Retries miss the mistakes that look right. Second opinions share the first opinion’s blind spots. Benchmarks cannot tell anyone their odds. It would be easy to conclude that nothing works.

Something does. It is simple, it works with the machines that exist today, and no AI vendor will ever mention it. Vendors are paid by the token, by how much their models produce. The remedy is to produce less.

Build less. Check what gets built.

It sounds like the opposite of ambition, and nobody will find it exciting. But if the goal is dependable software, rather than simply more software, the arithmetic leaves no other approach standing.

A project that holds its count of parts low enough for each to get real checking, however many more the agents could produce, turns every percentage point of reliability into real odds.

The extra reliability does not have to wait for a better model. It comes from checking, and holding the count is what makes real checking possible. Earlier, the question was who, or what, could check everything the agents produce, and the honest answer was that nobody knows. That is a question about sprawling projects. A project kept small is a different matter. It can afford the checks a sprawling one cannot: checks that fail differently from the thing that did the work. That means checking the specification itself for ambiguities and mistakes, running the code against real inputs, testing the code against what the specification says rather than what it happens to do, and putting the result in front of the people it was built for. Checks like these are slow and expensive, which is why they cannot keep up with a flood of parts. What makes them work is that they fail differently from the agents. A small project does not need many of them. A few percentage points of reliability can be the difference between a software system that almost never works and one that works more often than not.

Holding the count also changes what better models are worth. When the count is allowed to grow, every gain in the models is spent on more parts, and the odds never move. When the count is held, every gain goes straight into the odds. The same improvement that buys nothing for the project that keeps growing buys a great deal for the project that does not.

The oldest way to hold the count is not to build what already exists. Software in wide use, whether sold by a vendor or shared as open source, has already been run millions of times. It has met inputs no test suite would think of, from people with no stake in believing it works. That is the strongest kind of checking that fails differently from everything an agent does, and it is the rule of three at industrial scale: the clean runs that prove a part is nearly perfect, piled up by the millions. A battle-tested part is still a part, and it still has to work, but it replaces many untested parts with a single proven one, and it frees a project to spend its own checking on the parts only it can build. No agent, however capable, can give a part a track record it has not had time to earn.

The conclusion is counterintuitive but hard to escape: a tool that makes adding things cheap pays off only for those who keep treating those things as expensive.

Most people don’t realize this, and so more software projects will fail than ever before. Although agents have given expert practitioners great leverage, the benefits accrue only to those who do what the agents cannot: decide what not to build, and check what does get built in ways that do not share the agents’ blind spots. As for novices, agents have given them a real ability to take on small projects, but also a false sense that anything is within reach, and with it the temptation to bite off far more than they can chew.