Nobody Has Actually Built a Software Factory

On August 27, Uber reported that more than 70% of its pull requests now come from agents. Posts like that one, “Running a Software Factory Efficiently at Uber Scale” land every few days now. “Software factory” is the phrase of the summer: Warp launched one on August 18, Factory is named after one, 8090 sells one, and that’s a partial list.

I spent August reading every first-party account I could find, at least 21 programs from Stripe to Meta to Ramp, looking for the factory. Not one of them built one. Every “factory” is the same thing: agents, running in parallel, on the process the company already had.

The last time a new kind of power showed up at a factory, everyone bought in immediately, bolted it onto what they already had, and waited thirty years for the payoff. The same thing is happening now, and the pull request is where you can see it.

The thirty-year retrofit

Electricity supplied under 5% of factory horsepower in 1899, 50% by 1919, 75% by 1929. For most of those thirty years, what a factory did with it was swap the steam engine for one big electric motor and keep everything else: the overhead line shafts, the belts, the machines clustered around the power source, etc. Ford rebuilt the factory floor instead. Between 1913 and 1914 he moved the work past the workers instead of the workers past the work, and chassis assembly fell from 12 hours 28 minutes to 1 hour 33 minutes, by his own account. The improvement is roughly 8x, and the electric motor wasn’t what produced it.

Someone rebuilt the floor this year, just not for code. Alejandro Maza Ayala, Chief Product and AI Officer of Kavak, the Latin American used-car marketplace, told a16z on August 10 that handing employees ChatGPT or Claude produces “no efficiencies,” so Kavak did the other thing:

“redesign your whole company around the agents,” starting with “rebuilding most of your APIs” so agents could use them.

Kavak now spends as much engineering effort on evals as on the agents, and the agents handle 96% of customer interactions and 95% of transactions. He says they convert 2.1x better than the human sales team did. In December, with tens of thousands of agents already running the business, Kavak tore the working architecture down and started over.

Now read what “software factory” means in 2026, in the words of the people selling one. Zach Lloyd, Warp’s CEO, launched Warp Factories on August 18 with this: “a cloud software factory is an automation loop around the SDLC, where cloud agents triage, spec, implement, review, verify and monitor work. Humans stay in the loop at key decision points.” Factory, the company, describes its product as something that “runs delivery inside existing pipelines through ready-to-use GitHub Actions and service accounts.” Kyle Daigle, GitHub’s COO, introducing the surface that five frontier labs’ agents now route through, made it a promise:

“You’re still working with the primitives you know — Git, pull requests, issues — and using your preferred compute, whether that’s GitHub Actions or self-hosted runners.” — Kyle Daigle, COO, GitHub

Around the SDLC, inside existing pipelines, still the primitives you know. Every one of those descriptions keeps the pipeline that was built for people and adds agents to it.

Humans are reading 1,300 pull requests a week

The operators are the ones to watch, because they’re the only parties who could have rebuilt their own floor and chose not to.

Stripe shifted the programmer to the approve button. By February, over 1,300 pull requests a week were, in Stripe’s words, “completely minion-produced, human-reviewed, but containing no human-written code.” I take that as a design decision. No human writes the code and every change still carries a human signature. Nobody’s writing, but everyone still signing off.

Anthropic’s pipeline has the same structure. Jason Clinton, Anthropic’s Deputy CISO, described the company’s own pipeline in July: “More than half of all code is being merged by our internal version of Claude Tag while human engineers focus on directing, setting intent, and owning final approval.” Humans are still the gatekeepers for half of the merged code.

Warp, whose product is the factory, grades its own by human touchpoints. Its core factory metric, per its August 27 post, is “average human touchpoints per PR.” It automates about 30% of its own tasks, and its self-improvement loop ends where everyone’s does: “Humans review those suggestions as PRs on the factory definition and merge improvements.”

Every operator and vendor that I reviewed delivers agent work as a pull request for a person to read. Uber’s 70% ships, in Uber’s words, “with human reviews/escalations.” Ramp’s engineers said why when they refused to give their agent a shared service account: “You do not want to knowingly create a vector for unreviewed code to go into the codebase.”

The approach works. PostHog went from 1,441 merged PRs in January to 4,725 in June on roughly 10% more engineers, with a revert rate of 0.2%. Those are agents on an ordinary web codebase, and it is the only escaped-defect number anyone in my research publishes. The 1899 retrofit produced real output too.

It is also the ceiling. None of these companies took the human out of the loop, and the next two sections are about why.

The agent writes the test that says the code works

Maisem Ali at exe.dev named the problem on August 27:

“But the agent was still grading its own work. If it misunderstood what I wanted, it could build the wrong thing, write tests for the wrong thing, and then confidently tell me that everything passed.” — Maisem Ali, exe.dev

The agent that writes the code also writes the test that says the code works. Will Wilson, CEO of Antithesis, told Ron Minsky of Jane Street on Signals and Threads in March what happens when the task gets hard enough: “eventually it deletes the tests or eventually it makes the test pass in some trivial way.” Minsky: “I do think this is getting a little better, but the phenomenon is still very strong.”

The industry’s fix is a second model that checks the first. Shopify ran that experiment and wrote it up on July 29: in one audit a model flagged 30+ security holes, and after verification every one was downgraded, dismissed, or reclassified. Zack Deveau, who built the harness, put the cost plainly: “noise sent to a developer is worse than no finding at all.” A model judging another model just scales the volume of findings without improving their accuracy.

In July, Anthropic made Claude go back through its own outages and asked it how many of outage-related bugs its automated checks would now catch. The answer was about a third, the other two-thirds still would have shipped. This is the best-funded verification program in the industry grading itself. Two thirds is why a human still signs the merge, because nothing upstream of them can actually be trusted to reject a change.

Where the human is already gone

There is one kind of place in the research where the human is already gone.

Meta’s KernelEvolve writes GPU kernels, and its gate isn’t a human reviewer:

“Every generated kernel passes through a rigorous validation pipeline that checks both correctness — bitwise accuracy against reference implementations — and performance.” — Meta KernelEvolve team

The output matches the reference or it does not, and the agent can read that verdict without being able to change it. That is enough to close the loop. Agent runs post-train specialized models with measured kernel speed as the reward, and the system gets better with nobody in the loop. The outcomes: 100% on KernelBench’s 250 problems, 160 ATen operators at 100% correctness across 480 configurations, more than 25% training throughput on MTIA, more than 60% inference throughput on Andromeda in production.

Baseten, a startup, published the same shape on August 28: kernels “identified, proposed, and implemented entirely by our agentic framework,” kept only if they “pass correctness and end-to-end performance checks,” now in production with latency down 42.3% on one image model.

Nicholas Carlini, a researcher on Anthropic’s Safeguards team, got the same result for the same reason when sixteen agents wrote a C compiler with a 99% pass rate on most compiler test suites: “It’s important that the task verifier is nearly perfect, otherwise Claude will solve the wrong problem.”

Kavak measures a conversion rate, Meta measures bitwise accuracy, Baseten measures latency. In all three the agent can read the score without being able to change it.

Now look at what the factories call “self-improving.” Uber’s post files its loop under future tense: “We are working on an automated way to record papercuts from agent skill executions and auto-generate skill updates from the collected traces.” Kiro’s post titled Continuous Prompt Evaluation, August 21, discloses its author in the body: “This article focuses on evaluating human-authored system prompt and configuration changes.” Warp’s loop, above, ends at a human merging a PR.

The split is consistent across the sample. Where correctness is a number, the loop closes and the human leaves. Where correctness is a review, the loop ends at a person. Self-improvement looks like a consequence of having a machine-checkable verdict rather than something a team adds to a pipeline.

2,000 years of simulation a day, and no agents on it

The implied move is to make application code a place where correctness is a number. Zero of 13 operators did it. Not one changed the language, the runtime, or the codebase the agents write into. Stripe’s monorepo is heavily typed, and Stripe typed it years ago, by humans, for humans, and describes its homegrown libraries as “natively unfamiliar to LLMs.” Meta, the largest operator in my research, went the other direction. Mark Harman, a research scientist there, argued in February that agentic pace is “scaling the cost of false positives and test maintenance to breaking point,” and Meta’s answer is tests that “are generated on-the-fly for each code change and do not reside in the codebase.” That is a move toward fewer durable checks.

TigerBeetle, a financial database that runs 2,000 years of simulated testing every 24 hours against its own code, is on the opposite end. Its style guide requires two assertions in every function, and its simulator, extended again on August 20, can replay any failure it finds on demand. It has the densest set of machine checks of any codebase in the research, built for humans doing safety-critical work, years before agents.

And there’s no agent layer on it. No `AGENTS.md`, no `CLAUDE.md`, no agent lane in CI. On August 26, one day before Uber’s post, the maintainers closed an agent-assisted contributor’s open pull requests with one sentence: “we’re not accepting agent driven contributions at the moment!"

If any codebase on earth could let a machine decide whether an agent’s change is safe, it’s this one. 2,000 years of simulation a day is more scrutiny than any reviewer will ever give a diff. The maintainers didn’t run the agent’s code through it. A person rejected it by hand, and they did’t give a reason why.

So the two halves sit apart. Uber, Stripe, and Anthropic run agents on pipelines built for people. TigerBeetle has the machine checks and no agents. Nobody has combined them, and what a combination would require is not a mystery: application code where a wrong change fails to compile instead of failing in review, one place to commit so the bug where the event fires without the row can’t be written at all, and a core with no wall clock and no ambient randomness, so the whole application replays from a seed. On that floor an agent’s change runs through thousands of simulated crashes before a person opens the diff, and the agent improves against a verdict it can’t edit, the way Meta’s kernels do.

Every stack you have worked on was built to fit inside a human’s head. Code a human can understand is code a human can approve, and that is the review gate, written into the source. The floor I am describing gives that up on purpose. No engineer can hold it, and none needs to, because the simulator explores more states overnight than a reviewer will consider in a career. Only the agent can say no.

Agents won’t build that floor on their own. They’re trained on the code humans wrote, and the code humans wrote is the bolt-on code: a service around a database, extensive tests beside the logic, a review at the end, etc. That’s just how software has always been built. Left alone, an agent will rebuild that floor for you in whatever language you hand it. Getting the other kind takes a compiler and a simulator that refuse the usual shape, and someone who designed the floor that way before the first agent touched it. The checks have to exist before the agents arrive.

Which number is it allowed to fail?

The only places anyone has built something recognizably AI-native are the places where correctness was already a number. Compilers and kernels. Everywhere else, the factory is a fast way to fill a review queue.

David Crawshaw, who co-founded Tailscale and now exe.dev, got closer than anyone. His line from June: “With agents, CI is useless, the merge queue is vastly superior.” His company merges without peer review and runs the whole test suite synchronously in the queue. He changed when the checks run. The code they run on is unchanged.

Next time someone talks about their software factory, skip the throughput slide. Ask which number the agent is allowed to fail, and who owns it.