A few weeks ago on Twitter I started seeing more and more people proudly posting that they’re taking a sort of ice bucket challenge and deleting all of their unit tests.
Proudly proclaiming that they’ve deleted 800k lines of unit tests from their monorepo with claims that the tests lock in slop, slow agents down and burn tokens. This may just be engagement bait or Sherwood might really believe they’re doing some good here. My team knows that any code that gets committed gets owned. I’d love to know how a codebase grows 800,000+ lines of tests nobody wanted. That seems like an interesting post I’d like to read. Last week I wrote about “multi-agent reviews with reviewbot” and the goalposts have moved from don’t review code to don’t bother testing it either.
Kun Chen jumped into the discussion as well with a DeepSWE run comparing tasks with and without tests. Without tests, agent time dropped 6% and spend dropped 9%.
Lower numbers might look compelling but it’s a benchmark of one-shot tasks and not a test of whether future changes still match the intent. Kun even says that “both the implementation and the tests were simply the agent’s interpretation of our intent,” which points to nobody reading the tests, not to the tests being bad.
Many others jumped into the fray with hot takes like Nate Berkopec who wrote that agent-written unit tests are “terrible, at best just doubling total LOC.” Ansh Nanda’s AGENTS.md in which he’s written:
Looking at these rules the 3rd one is discipline to write the failure modes first. An agent will read that and unless the harness uses more than one agent, the same agent that writes the failure modes will grade them. A lot of agent-written tests can be worthless and even more so if nobody ever bothers to read them.
If the tests written by an agent were truly adding no value at all they got that way because whoever prompted them into existence never read them, and they got merged in anyway. The tests were never the problem. Nobody reading them was. Payment is coming due for teams that let models guess intent from prompts instead of writing the intent down.
The standard reasons for writing tests:
1. Catch regressions. A test fails when a later change breaks behavior that used to work. It’s the most common reason given, and the safety net that lets a team change code without fear.
2. Make refactoring safe. You can restructure code and know you didn’t change what it does. Martin Fowler’s Refactoring treats a solid test suite as a precondition for refactoring at all.
3. Fast, precise feedback. A unit test runs in milliseconds and tells you which piece broke, not just that something somewhere did.
4. Document intent. A test shows how the code is supposed to behave, in a form that can’t drift out of date without failing. It’s documentation that runs.
5. Pressure toward better design. Code that’s easy to test in isolation tends to be loosely coupled. This is the argument from Kent Beck’s test-driven development: write the test first, and the test shapes the design.
6. Find bugs earlier, when they’re cheaper. This rests on the idea that a defect costs more the later it’s found. It’s widely cited, but the exact numbers are disputed.
7. Confidence to deploy. Continuous integration and frequent releases assume an automated suite tells you the build is safe to ship.
A test is there to protect the future. And a properly written unit test is the cheapest method of doing that. They run in milliseconds, offline and must give the same answer every time. When a unit test fails it’s clear what part of the system broke because it’s usually a single function. When an end-to-end test fails it points at everything.
Online I’ve seen instances where developers are proclaiming that all of their tests will be end-to-end. So we’re trading tests that are cheap with tests that are slower, need a running system and fail intermittently. An end-to-end test belongs at the top of the pyramid and when you move everything there, it gives you less coverage at a higher cost and topples over.
Reading a test isn’t the same as reading code. With code you can ask “does this work?” With a test you ask “would this notice if it broke?” Getting an agent to write that test takes more than one prompt, and it takes a reader to validate it.
Here’s what I look for.
Break it on purpose. Change the code under test: flip a condition, return the wrong value, then run the suite. If nothing fails then the test isn’t protecting anything.
Behavior, not implementation. A good test checks what someone asked for: the total includes tax, an expired token gets rejected. A weak one restates how the code does it: these three functions get called in this order. The second kind breaks every time you refactor and stays green when the behavior is wrong. Agents write a lot of the second kind, because they wrote the code first and are describing it back to you.
Watch the mocks. If everything interesting is mocked, the only thing being tested is the mock. A mock can’t stand in for an integration test.
Treat test edits as suspicious. Don’t let the agent change a failing test on its own. I’ve had agents delete tests to get everything to pass. It’s up to you to decide if the test was wrong and for the PR to be clear about your intention with the team.
Let agents review the tests too. I don’t read every line of every test, and neither will you. Point a reviewer at the tests specifically, with a prompt that asks whether each one would fail if the behavior broke, and use a different model than the one that wrote them. That’s part of why I built reviewbot: a second opinion that isn’t the author.
The above is part of what code review is supposed to be. The difference is volume. When an agent writes a hundred tests in an afternoon, reading them is the job, not an afterthought. A Harvard study1 of 718 companies found that after teams adopted coding agents, the share of pull requests needing changes nearly doubled. The reviewing didn’t go away. It became the work.
Let’s imagine that you wanted to disrupt an entire industry that was over 130 years old. If you were making a big bet like that you would probably pour as much money as possible into ensuring that you were able to change consumer behavior by making the experience better and incentivizing use. Uber’s S-1 discloses more than $1.8 billion in driver payments above trip revenue from 2016 to 2018, alongside separate rider promotions—subsidies designed to influence whether people used the service.
Frontier labs are doing the same today. Cracks are already starting to show. OpenAI added a $500 Pro plan, and its $200 plan reportedly now covers about half of what it used to. Capacity is priced below cost to win users and then repriced once you’re locked in.
Assuming that tools that you rent at a subsidized price are always going to be available, affordable and unchanged is probably not wise. Deleting unit tests out of your code because you let the agent run wild creating validations that were unnecessary doesn’t mean the practice is wrong, it means that you stopped paying attention. Tests tell you and your agents (frontier labs or open weight models) what the app is intended to do in the one language computers understand: code.
The commit, the PR, that code that you pushed into GitHub has your name on it and that needs to mean that you stand behind it. It should cover the requirements for that application, and be verifiable with a test. The understanding you’ve outsourced is rented at prices that probably won’t last, from labs that are still learning to contain what they’ve built.2
In the last 18 months code generation has gotten incredibly cheap, and probably more than it should be: today’s subscription prices are almost certainly below what it costs to run. We’re living on borrowed time and we’ve outsourced the understanding of products we’ve built to a model we rent.
Tests are a large part of how you keep that understanding in the organization. When they turn out to be bad, the answer isn’t to delete them. Read them. Use the agents and your own eyes to make sure every test checks something real, so the next change, yours or the agent’s, can’t quietly break what you already shipped.