How to effectively debug when you no longer understand the code

Disclaimer: While the content of this article might be Agentic coding, this post was entirely written by hand, except for some parts where I directly quote LLM generated text, which is attributed as such. I did, however, use Claude to generate all visualizations for this post, which I feel deserves to be stated upfront.

This is quite a long post, but you can think of it as consisting of four parts (and the navigation panel to the right of the page will display the table of contents and help you navigate to relevant sections).

“The Tale of Three Bugs” introduces the tale of three bugs I found myself having to fix at work with the help of agents.

“Do we still need to understand the code?” is self-explanatory.

“Agentic Paradigms for Debugging Unfamiliar Code”, is perhaps the crux of this entire article. It introduces a series of general techniques I’ve been using that enables me to work effectively and methodically with agents to debug unfamiliar code.

“Retooling the Agentic Factory Floor For Autonomy And Accountability” provides a framework for evaluating tradeoffs between optimizing for agents and for humans when these two goals might be at cross-purposes, identifies missing pieces of existing “software factory” implementations that allow for fully autonomous debugging, and details the paradigm shift in thinking required by those in the “agent-native” vanguard.

I understand not everyone might have the time or the patience to plough through this, but might still be interested in the main takeaways. You might be best served by asking an AI to summarize this post for you. If you do, however, still maintain a bias for human writing, enjoy this at your leisure.

Diving Into A Brave New World, Eyes Wide Shut

The ability to work with large amounts of unfamiliar code productively without necessarily understanding the specifics or reading the code can be quite a superpower, one that enables engineers to tackle problems way out of their depth.

I consider this an emergent skillset that’s different from coding or traditional debugging, but like any other skill, it’s one that can be honed with enough effort. While I’m no leading expert in this area, the general contours of how these skills can be actively cultivated are beginning to stand out in sharp relief.

By putting my own experiences in writing, I’m hoping to hear from fellow practitioners their thoughts on this topic.

The Tale Of Three Bugs

I often find concrete examples involving real, production software missing in most discourse around agentic coding. Loath to repeat the folly, I want to start with the tale of three bugs I had to fix at work over the last couple of months: the latency bug, the memory leak, and the “hang”.

All of my beliefs around agentic software development detailed in this article are rooted in the experiences I’ve forged in the petri dish of working on codebases and services that are nearly a decade old, written largely using bespoke tools and libraries with a smattering of open source components.

The three bugs discussed here are interesting because while they have little in common on the surface, they are good vehicles for demonstrating different approaches I had to use to steer an agent towards the right solution. The bugs differ from one another in terms of:

• how familiar I was with the code where the bug was reported
• whether the bug manifested in new code or was a regression in existing code
• whether I had source code access to the component where the bug originated
• whether the bug was reproducible
• how long it took to identify the “root cause” and apply a fix
• how the fix was verified

These are important axes of differentiation because they influence which approach might be best suited to debug any given type of bug.

What the three bugs do have in common is what it took to keep the agent’s conclusions honest in each case, and the role of the human in agentic investigative loops.

Control, Variable, and Signal

There are three terms I will be using in this post that some readers might be unfamiliar with. I want to begin by establishing what these terms mean in the context of this post.

The variable is the broken case where the bug shows up.

A control is a case you compare the variable against. Ideally, it’s identical to the variable except for the one thing you suspect is causing the bug. For example, if enabling a feature flag causes a bug, the ideal control would be running the same code with the feature flag disabled. The most obvious candidate for a control is just an instance of the code that is working correctly without the bug, though as we’ll see later, sometimes, the most useful control is a case that also fails, just in a different way from the bug under investigation.

A signal is any data point that might identify or explain the bug. It could be a log line, a latency measurement, an error, a call stack, a counter, the value in a register, and so on.

Signals are important because they are the best entry point to any investigation for an agent. Every hypothesis an agents thinks is worth pursuing should start from a high-fidelity signal.

The Latency Bug

This was a bug in a new feature added to a storage service. The feature added about 9000 lines of new code across two repos, and was written entirely by agents from start to finish in a little over a day. By the end of the second day, I had a fully working prototype ready.

This was by no means whatsoever a “vibe coded” feature. Au contraire, the feature was very meticulously developed. I started by giving Claude a detailed description of the new feature, along with a comprehensive set of instructions on precisely how I wanted it implemented, including APIs, layering, telemetry, edge cases, testing, error handling, and so forth.

Once I had this fully working, I set out to break this ~9000 LOC prototype into 10 separate patches, ordered in ascending order of risk level, so that I was merging discrete increments into the main branch, rather than land ~9000 LOC in one fell swoop.

I’d personally read every single line of code in every single patch, starting from the lowest risk one to the final, highest risk patch. I’d subjected all of the patches to multiple rounds of reviews from a suite of subagents. I’d made a lot of additional changes to add polish and clarity to the patches. All of these patches were also reviewed and approved by a human coworker.

We run hundreds of storage servers across multiple Kubernetes clusters and namespaces in different regions. Our server side deploys first go out to a pre-production cluster which has been typically pretty good at detecting a wide genre of bugs: crashes, data corruption bugs, data integrity bugs, and so on. The promotion of this new feature to the pre-production environment went without incident.

When it came to the production deploy, the feature flag was first enabled on the server. The server side feature flag enablement is a no-op unless the feature flag was also enabled on the client. The client side story is more complicated as the client runs on hundreds of thousands of machines in various AZs, regions, data centers, etc. We typically start the production rollout with the smallest client deployment, which covers about ~5000 hosts in a single data center.

Once the feature was enabled on the client-side, I began monitoring all relevant telemetry graphs: request rate, success rate, errors, latency, etc. It can sometimes takes a few hours for issues to manifest in the graphs. But within about 10 minutes after the feature flag was flipped on the client, the P99 latency graph on the client looked like this.

I immediately correlated this with the server side latency graphs, which looked like this:

From both aggregate latency graphs, it very much looked like the server was taking much longer to respond to requests when this new feature was enabled. Now, the challenge was to figure out what caused the server’s performance to regress so severely.

I did what everyone does these days, pointed Claude at the latency telemetry, and asked it to debug the latency regression. This wasn’t some vague request either. I gave Claude specific instructions about this regression:

• I asked Claude to check if the latency regression was reproducible locally, either in a unit test or with a local client and server
• I asked Claude to take a close look at the client and server production configurations for various tunable attributes
• I gave Claude access to all previous performance related investigations and incident postmortem documents pertaining to this server
• I gave Claude read-only access to the telemetry system to run additional queries to gather more data
• I asked Claude to specifically look at the git history of the entire project and check if any performance anti-pattern that was previously fixed had been mistakenly reapplied again in any of the ~9000 LOC
• I asked Claude to have an independent subagent validate any theories it might come up with

Claude churned for about 5 minutes, and came up with an answer. It had determined that the “root cause” lay with how the server was loading a TLS certificate when connecting to a remote. The TLS cert loading logic was a new functionality that had been added as a part of this new feature, to make sure neither the server nor the client ever used a stale, expired cert. Claude’s deduction was that the cert reloading was happening not just once during the initial TLS session establishment, but once during every single RPC call the server made over the established TLS session to its remote. Claude determined that the fix was to make sure we’re not reloading the cert per RPC call.

In the 9000 line patch, I’d certainly taken the effort to make sure the cert reloading was happening correctly, and had tests that detected the certificate expiry scenario. But I hadn’t done any performance testing around this, so Claude’s initial diagnosis that the latency regression was caused by suboptimal cert reloading sounded plausible enough to me.

Claude proposed a fix for this, but before applying the fix, I wanted to make sure that there were no additional regressions that would in turn take additional cycles to discover and fix up. I was also concerned about the overall operational impact of the latency regression on the existing production deployment. From experience, I knew that when distributed servers slowed down for whatever reason, latencies aren’t the only thing impacted. Slow responses can cascade into excessive client retries, load shedding, failed health checks, and server restart loops, and alerts for these can take hours to fire.

I knew from experience that there were things that weren’t neatly captured in any telemetry that were often leading indicators for future service availability issues. One such signal was the client TCP session establishment time. If a server was in its steady state, it typically took a client a few tens of milliseconds to establish a new cross datacenter TCP session with the server. However, if the server was slow, connections could back up waiting to be accepted and new handshakes could take much longer. I knew measuring the TCP session establishment time on the client would give me an idea of how bad the problem was. As this wasn’t reported as a part of our normal metrics, I had to measure it manually.

I asked Claude to ssh into one out of the 5000 production client hosts on which this feature had been enabled. I told Claude to use tcpdump (and if that wasn’t available on the host, to use whatever equivalent tool might be available) to measure the time taken by the client to complete the TCP handshake with the server exhibiting the latency regression. As a control, I also asked Claude to run the client with the feature flag disabled for this new feature against a different storage server deployment which also had the feature disabled, and measure the TCP handshake time, which I needed for a baseline.

For the sake of completeness, I also told Claude to use any other tools that might be available to it on the host, to see what else might be different between the control (the case where the feature flag was disabled on client and server) and the variable (the case where the feature flag was enabled on the client and server). Specifically, since the client and the server used the same cert reloading logic, I asked Claude to check if the client was also reloading the cert when making every RPC call to the server, by using whatever tool might be available to it to monitor a running process.

Claude did as asked, and came back to me with some surprising findings. The first finding was that the TCP session establishment timings didn’t differ between the control and the variable. The second finding was that the cert was only being loaded during the first TLS session establishment, and not for every RPC call. This was a key finding, because without this piece of data, I’d have shipped Claude’s initial suggestion, which would’ve proved to be wrong.

The third and most important finding came from my request to Claude to “use any other tools available on the host, to see what else might be different between the control and the variable”. Claude monitored the DNS lookups the client was making, and spotted that the control’s lookup returned around 5 addresses whereas the variable’s lookup returned around 400 addresses!

This was the real problem. Claude’s initial claim that the latency regression was due to the server reloading its TLS cert too often when connecting to its remote proved to be a red herring. The P99 latency had regressed because something about client side service discovery had regressed. Claude then used this evidence to look more carefully into the code to see what was different between the control and the variable when it came to shard discovery and routing, and determined that the new mTLS codepaths introduced in the 9000 line feature were not using the sharded client.

The fix was literally 1 line: using ShardedDatabaseClient instead of DatabaseClient.

Server side latencies shot up, not because there was anything wrong on the server, but because the client wasn’t routing requests to specific shards that already had the data but was instead routing it to any random pod, which would then have to fetch the data from the right shard (in case of reads) or route the request to the right shard (in case of writes), which led to increased server side tail latencies for both reads and writes. Even if this client side regression only impacted the smallest 5000 host cluster, that was enough to upset the steady state of the sharded server, whose P99 latencies shot up concomitantly.

Claude, more by accident than by design, stumbled upon the DNS lookups the client was making and then correctly deduced that breaking client side routing was what caused the latency regression. I would’ve never suspected that DNS would prove to be the smoking gun here. What I was hoping to find was the initial TCP session establishment taking longer, resulting in potential head-of-line blocking, and all the attendant joys that entail. What I ended up discovering was, well, the true “root cause” of the P99 latency regression, as well as the right fix for the problem.

There’s a lot to extrapolate from this example, but I want to hold off on drawing too many conclusions just as yet from this solitary example.

I have two more fascinating debugging stories to recount before I draw any conclusions. But I do want to fend off a few obvious questions folks might have at this point:

Didn’t unit tests catch this?

The sharded and non-sharded clients are both valid.

We only use the sharded client when connecting to a remote sharded backend. We also have non-sharded storage backends (a sqlite backend, an in-memory storage backend, etc, none of which require any kind of sharding).

Using the non-sharded client compiled just fine and passed all local unit tests.

Were there no integration tests to guard against sharding regressions?

There were integration tests, but none that tested that this new feature used the sharded client to open new TLS connections to the remote storage service.

Why wasn’t this caught in the pre-production environment?

The pre-production cluster is much smaller than the production cluster.

The pre-production environment runs a 2 shard configuration with 1 replica per shard, whereas the production servers run 5 shards with 80 replicas each.

The client was only looking up 2 addresses in the pre-production environment and about 400 in the production environment. The bug only truly hurt latencies at production scale.

The Memory Leak

It’d be interesting to contrast the latency bug to another bug I solved using a diametrically different approach than the one chronicled above.

A few months ago, I got a bug report from a user about a memory leak in a tool running in a Docker container. The container was being OOM-killed when the user did not set up the right authentication credentials for a remote service. This was very odd, because a failure to set up correct authentication credentials would result in the user seeing an authentication error, not being OOM killed.

At the heart of this bug is a state machine that deterministically tracks the existence and validity of various authentication modalities available to the client. The state machine is recursive in nature, and depending on what state it’s in, takes different actions to procure authentication credentials. The state machine terminates if there are no valid authentication credentials available. The state machine also does a lot of bookkeeping like keeping tabs on authentication token and TLS cert expiry, renewing tokens or certs when needed, retrying transient failures and more. As you can tell, this is way more complex than simply working with a static API_KEY as the authentication credential. Notably, the state machine was designed to deterministically terminate, not recurse forever.

What went wrong?

The bug report from the user came with a wealth of useful information, including the version of the tool where the memory leak was first seen, the last known good version of the tool that did not produce the memory leak, detailed steps on how to reproduce the memory leak in a Docker container, and the output of heaptrack which showed me where roughly 50GB of allocations had happened on the heap before the OOM-kill had happened. With all of this information (especially with detailed steps on how to reproduce the bug), I thought Claude would “one-shot” this investigation and get to the bottom of the bug in no time.

Claude’s very first suggestion was that the memory leak was due to the client generating authentication tokens correctly which the remote server was rejecting, which led to the client retrying the request 137K times before it got OOM-killed. Claude claimed each of these attempts appended a JSON event to an in-memory log and that the log had grown large enough to use up 50GB of memory. As evidence, it cited the state machine reaching a particular state, and as a solution came up with a detailed plan to … bound retries.

I knew that what Claude cited as “evidence” could never happen. I knew this with a deep-seated certainty, because I’d manually written every single line of the state machine myself back in 2025. The state machine itself was a part of a much larger feature which took several weeks to author (which, just a year later in 2026, feels like an eternity), but I know exactly what states can be reached, when and how.

When I pushed back, Claude admitted its mistake, and went into another investigative sprint. It came back around the second time with this:

the range's amortization + telemetry work is what converted a latent infinite
loop into a multi-GB OOM.

Claude was wrong yet again.

While I was aware there’d been some recent work done to amortize certain API calls, amortization was disabled by default. A user had to opt-in to enable the amortization feature, which the user who filed the memory leak bug had not.

When I pointed this out to Claude, it came back with:

You've found a real hole in my theory.
Let me confirm the gating precisely, because it splits two different
mechanisms I was sloppily lumping together:
- Engine amortization is gated → opt-IN , default OFF.
- Process-wide cache is gated → opt-OUT, default ON.
The heaptrack frame is runStateMachine on line 562, the plain internal 
recursion, not dedupedRunStateMachine on line 683. 
That means my "engine amortization caused it" claim is wrong
for the default config, and I need to retract it.

Claude then went on to propose four additional hypotheses, suggesting that it would need to bisect 24 commits to validate the additional theories. It asked me whether I’d like it to proceed.

Instead, I suggested it take a different tack:

Instead of bisecting, how about this.
this leak is caused when no valid auth credentials are supplied. 
this only happens with dynamic modes of authentication, when 
a token/cert is minted successfully but the server rejects it, 
OR the token/cert is not minted at all.
trace the code to see what happens in both these cases, and when we might run
into an infinite loop. carefully trace all the code, and see wht could lead
to the heaptrack output

Given adequate directions, Claude got it right the fourth time around.

The state machine I’d designed was meant to be single-use only. A recent change (a process-wide cache that was on by default) that shipped with the amortization work violated this design principle by keeping the state machine alive for the lifetime of the process. To cite Claude’s correct analysis (after three previous failures):

#1431's process-wide static engine cache changed the state machine's lifetime
from per-session (discarded after exhaustion) to per-process
(pinned and re-entered forever), turning a state that used to be entered
exactly once into one that gets re-entered on every retry.
Same "no auth token" input; the old version threw the dead state machine away,
the new version keeps feeding it back into itself. Swift async code has no 
tail-call optimization, so every one of those recursive calls kept its frame 
alive, and the frames piled up until the container ran out of memory.

This really should’ve been an “easy mode” bug, given it came with steps on how to reproduce the bug and a heaptrack output, but Claude still needed directions, guidance, and pushback before it could nail down the root cause.

I do want to answer some obvious questions folks might have:

Why didn’t you ask Claude to reproduce the bug?

The bug report had explicit instructions on how to reproduce the memory leak, so I thought Claude would start there without me explicitly telling it to do so.

I was as surprised as anyone else when Claude did not first try to reproduce the bug, but instead offered a wrong analysis based on reading the code.

Were there no tests to make sure the state machine was a single use construct?

The state machine has a lot of tests, but none that enforced that it could never be used in a process wide cache, where the state machine would be reentered multiple times over.

I simply did not anticipate someone would try to use it that way.

The Hang

The “hang” might be the most esoteric bug I’ve ever had to debug in over a decade of being a software engineer.

It’s also a bug I never would’ve managed to get to the bottom of if it weren’t for a tool like Claude. It goes further than any previous anecdote to hammer home the point that agents enable you to solve problems that require you to sojourn way out of your depth, as I did in this case.

It started with a slew of bug reports being assigned to me about mysterious timeouts seen in a daemon that ran on the desktop. This daemon connected to a Websocket based remote service that my team owned, and exposed the data fetched from the remote service as a virtual file system.

I personally have no experience building or debugging desktop software, especially not desktop software that ran on experimental, unreleased versions of the operating system. But this daemon was using a client library to the remote service that was maintained by my team, and hence the debugging responsibility fell to me.

While the bug reports from users notified me of frequent timeouts, there were no reported crashes or other errors logged by the daemon. The timeouts themselves didn’t originate from the client library my team owned, but originated from the virtual filesystem layer when a request to the remote system didn’t respond within 60s.

My initial hunch was that the remote server wasn’t responding to requests within 60s. Usually, a read or a write operation only takes a few milliseconds. 60s timeouts are quite rare, but not entirely unheard of, especially during periods of server side CPU throttling, or during event loop stalls due to some blocking code getting in the request path. These instances of server slowdown were rare, lasting no longer than a handful of seconds. I correlated the server side latency telemetry with the timestamps when the VFS timeouts were seen, only to see that there was nothing anomalous recorded on the server when the client was seeing timeouts. The problem was not on the server. It was with the client.

However, this was not a bug that was reliably reproducible on the client. The timeouts themselves were intermittent. The only evidence I was left with was various System Diagnostics outputs taken by the users as a part of their bug reports. For the uninitiated, the System Diagnostics (or sysdiagnose) gathers system-wide diagnostic information helpful in investigating system performance issues. Usually the system logs found in the sysdiagnose give you a good idea of what went wrong with an application running on the system, but in this case, apart from the timeouts, as stated previously, there was nothing reported in the logs.

Couldn’t I just add more logging?

Sure, but this daemon was not owned by my team. It just linked a library my team maintained. The daemon also has a slow release cycle, which meant any additional logging would take weeks to reach the end users.

Many of the users who had complained about the timeouts had also resorted to restarting the daemon when they saw the timeouts, which wiped clean any evidence I might’ve been able to gather from a daemon caught dead to rights, like the daemon’s call stack profile. Without logs, or profiles or call stack samples to tell me what was obviously wrong, I was left to piece together the source of the timeouts from miscellaneous morsels of eclectic and unfamiliar files that could be found in the sysdiagnose, most of which I’d never opened in my life.

My first hunch was to look at packet capture output, to see whether this daemon was even successfully establishing a TCP connection to the remote server, or whether the timeouts being observed were due to some kind of packet loss, leading to the client simply never connecting to the remote server. Fortunately, a sysdiagnose output has a wealth of networking data available for analysis. Do I personally know how to interpret each and every piece of this data? Some bits of data (tcpdump, netstat etc), sure. All of it? Absolutely not.

But a model does, and if you give it access to all of this data and ask it incisive questions, it’s extremely adept at getting you precise answers. An inspection of packet capture data led me to conclude with Claude’s help that the initial TCP handshake was successful and was followed by a successful Websocket connection upgrade, but after that, no read or write requests flowed. The only requests that could be seen in the packet capture output looked like the periodic Websocket pings and pongs. This was a pattern that reliably repeated across most users who reported these timeouts.

During normal mode of operation, after the Websocket upgrade had been completed the daemon would proceed to authenticate with the remote server. For authentication to succeed, the daemon is required to present a valid authentication token to the remote storage service. It acquires this token by making a Mach IPC request to another authentication daemon running on the host. Once our daemon had this token, it would go on to establish an authenticated session with the remote server. Once an authenticated session was established, read and write requests could commence.

The fact that I was seeing evidence of successful Websocket connection establishments but seemingly nothing happening after that indicated that something was likely awry with the authentication.

Usually, authentication failures lead to errors reported in the logs. The failures are almost always due to users not being authorized to access the remote server, or lacking specific permissions. But in all of the reported cases, there were no logs whatsoever from the authentication daemon. I thought it was still worth a shot to escalate this bug with the team that owned the authentication daemon. However, that team cited the lack of any logs as proof that their component was not to blame. I was told in no uncertain terms that if there was something wrong with the authentication daemon, there would be logs explaining exactly what went wrong and why, and that the lack of logs meant that there simply was no authentication request made to their component.

Being rebuffed by the authentication team threw a spanner in the works. All I had left was a veritable mountain of unfamiliar data to trudge through, hoping to find a needle of elusive evidence in a haystack.

Debugging from this point involved spending the best part of a day wrestling with the diagnostics data. It ultimately took looking at the output of something called lsmp (which I’d never heard about before, and learnt was short for “list Mach ports”) and a lot of kernel level IPC data to conclude that while the authentication daemon accepted a request over XPC, it never called the completion handler, leading the client that sent the request to block forever waiting for a valid authentication token to be returned. Here was Claude’s summary of how lsmp data helped diagnose the root cause:

lsmp is to inter-process communication what lsofis to files. lsmp lists every process’s Mach ports, the kernel channels that XPC messages travel over, along with who holds which “rights” to them. The proof was a single line in a 30 MB file. When a client sends an XPC request, the kernel gives the server a one-time “reply ticket” that is destroyed the moment it’s used. Our daemon’s request was still unanswered 67 hours later. A reply ticket that still exists is, by definition, a reply that was never sent.

This crumb of kernel level evidence reliably showed up across multiple user reports. I was then able to file a new, comprehensive bug report with the authentication team again with the newfound evidence.

Make no mistake, finding this piece of “evidence” involved lots of false starts that necessitated plenty of about-turns, lots of validation of various hypotheses (both those proposed by the agent and proposed by me) which in turn required a barrage of repeated checks against the mountain of data, all of which was only possible because the agent made these checks relatively cheap and easy to run. Without the ability to constantly run these checks, this problem would’ve proved to be positively intractable.

Do we need to still understand the code?

Up until this summer, I stood steadfast in my belief that all code must be fully understood by the developer tasked with generating it, even — or especially — if they didn’t manually author any of the code themselves.

To be clear, this is a different claim from suggesting that the developer should understand the entire codebase they’re contributing to (which, for larger or legacy projects, can be a rather unrealistic proposition).

The ask was more curtailed in its demands: if you’re contributing a patch, however large or small, you need to understand what problem it addresses, how it works, and why it works the way it does. If you’re fixing a bug, you need to understand what caused the bug and how the patch addresses that. If you’re contributing a new feature, you need to have a good understanding of why you chose to solve it the way you did, how it fits in with the existing code, whether it meets the bar for reliability, performance, and security guarantees provided by the existing code, and so on.

In the pre-agentic coding era, this often required a developer to do certain things, such as, read relevant parts of the codebase, figure out how and where the new changes fit in, look at the existing APIs to analyze how best the code change should be written, figure out how best to test the patch to safeguard against future regressions, and so forth.

If you were a more senior engineer, your responsibilities ran a more panoramic gamut as well. You had to keep your eyes peeled for structural deficiencies and accidental complexity. You had to make sure discrete changes knit together into a cohesive whole over time. You had to see to it that interrelated problems got detangled when the opportunity presented itself. You had to keep an eye out to ensure the codebase’s overall topography wasn’t characterized by rolling hills of local maxima accrued over time as engineers favored point solutions over more holistic ones.

Agentic coding upends almost all of these precedents. Developers can (and do) now ship patches without putting in any work whatsoever to understand the feature or the bug. The results, from what I’ve seen, have been chequered, across orgs, and sometimes even with the same developer. While agentic coding can often “one-shot” bugs and features, I’ve also seen bugs “fixed” that require multiple rounds of back and forth to be fully resolved, with the developer admitting they didn’t necessarily understand what the patch really did the first time around.

In many respects, code not working as expected is the “happy path”, since this squeaky wheel is rest assured to get greased in due, if not immediate, course. More onerous is the torrent of complexity, most of it extraneous, that’s being conjured up in dizzying haste, as developers succumb to the sugar-rush of shipping an onslaught of features which bestows upon them a patina of hyper-productivity. Every organization is going to demand its pound of flesh in exchange for the eye-popping amounts of money being sunk into agentic tools. “Technical debt”, which even in the best of the pre-agentic times was never properly addressed, will continue to fall even more by the wayside.

These are inevitabilities, no matter how hard some might try to deny it, and I fear there’s no use crying over spilt milk anymore. What we do control as software engineers is how we adjust to this new reality.

So, given all of this, do we still need to understand the code?

For long, I’ve held it as an unimpeachable article of faith that having a strong mental model for how code works is absolutely necessary to be able produce — and more importantly, to be able to debug — high quality software. Working with a developer or a team who did not possess a deep understanding of their code or their systems unfailingly led to poor outcomes.

Recent experiences, however, have me reconsidering my own long-held beliefs in a different light.

Code is going to be written, read and debugged almost exclusively by agents going forward. Even if you, like me, find it hard to let go of old shibboleths like reading and understanding every line of code, the depth of understanding you glean after a perfunctory scan of the code is materially different than the understanding you had when you were manually writing code.

Handwritten code was a tapestry of a large number of conscious decisions on the part of the programmer. Even the minutiae (like the name of a file or the various fields that make up a struct or the signature of a method) required a certain degree of thought from the programmer. The mental model construction was a slow, synchronous process, inasmuch as there was only one line of code you could write at any given time. Subsequent bug fixes, performance improvements and the like presented additional opportunities for chewing the cud, and for the mental model to be revisited, corrected, honed, and polished.

That “slow burn” approach involved in the manual translation of high level intent into working code left the mechanics of the code etched in your memory in a way that is nearly impossible to replicate when an agent generates thousands of lines of code in a matter of minutes.

I do not possess the same degree of familiarity with agent generated code as I do with code I wrote myself; sure, I know the general silhouette of the design and the high level decisions but not every last intricacy. Even when I take the trouble to read every line of agent generated code, my understanding is not quite as deep or as comprehensive as it was with the “slow burn” approach.

And rest assured, it’s a racing certainty that my understanding is not deep or lucid enough to authoritatively push back against an agent when it’s categorically wrong. Which it is, a lot of times.

Any “skill” or process or tooling or software factory needs to make allowances for this fundamental fact.

This calls for new paradigms that will allow engineers and agents to ship and debug large amounts of code productively when the engineer might at best only have a passing understanding of the code, and at worst, might have absolutely no understanding of the code.

The desired end goal is to have a process in place that allows agents to debug any kind of problem with minimal to no human inputs. This might remain an ambitious and aspirational goal, but it should be the lodestar to teams building agentic platforms and tooling.

Agentic Paradigms for Debugging Unfamiliar Code

Agents can and do allow engineers to be orders of magnitude better and more productive if they set their hearts to it. I’d have never on my own ventured forth into inspecting kernel-level data that was required to correctly triage the third bug, but with a model at the ready to assist me, I was more than up to the task.

Did I understand every bit of analysis Claude presented, or every deduction it arrived at by examining the data (data that, I should remind you, I hadn’t the slightest clue how to interpret)?

No.

But my argument is that even when such a level of detailed understanding is missing, one can still squeeze enormous amounts of utility from a model, without risking being lost in the weeds. The “hang” was a truly wacky bug which required me to try to tease meaning out of kernel object and port data, but the underlying ethos generalizes well, and can serve as a template for debugging almost any problem with the help of a model, even when one is largely unfamiliar with the code.

Play to Agentic Strengths, not Human Strengths

One of the most obvious mistakes I made with the “latency” bug was to start by asking Claude to debug the problem as I personally would’ve as a human. Everything from trying to get Claude to reproduce the latency regression locally, to asking it to pattern match with previous performance incidents was inherently human-centric at heart. Humans tend to reach for certain tried and trusted techniques for debugging. It’s foolish to expect the same techniques to work well with agents as well, especially when our goal is to minimize human involvement in the debugging loop.

Agents are not humans. They’re far more capable than humans when it comes to many things, and we should be leaning into these strengths, rather than trying to get agents to replicate human patterns of problem solving which might be suboptimal in agentic contexts. If I were to list a key set of agentic strengths, it’d be:

• Very quick at processing and making sense of massive volumes of data
• Phenomenal at cross-referencing different sources of data from different layers (kernel, filesystem, packets, process state, thread stacks, etc)
• Broad working knowledge of almost every tool and data format in existence, which is far outside any one engineer’s wheelhouse
• Tireless, so every hypothesis, especially those too onerous to test by hand, can be tested as many times as needed
• Very adept at writing throwaway tooling (shell scripts, benchmark tests, performance tests etc) on demand

Agents also have predictable weaknesses. They often build baseless narratives out of thin or non-existent evidence. They can fixate on abstruse edge cases or walk past their own previous best findings.

A good “agent-native” debugging workflow is one that leans into agentic strengths and has guardrails against agentic weaknesses.

Code is an Unreliable Narrator

Speaking of leaning into agentic strengths, having an agent start an investigation by first reading the code is an anti-pattern.

In fact, reading the code first to formulate hypotheses about a bug should always be the very last resort of the agentic investigation process, not the starting point. An agent should first collect all available evidence about the bug, and only then formulate the hypotheses from the code that might best explain what’s being observed.

The reason being code is an unreliable narrator, and building hypotheses for debugging solely by trusting an unreliable narrator sets an agent up for failure.

I call code an unreliable narrator because code tells an agent what can happen, not what did happen. We want to start on factual footing, not speculative footing.

Another problem with starting debugging investigations by reading the code is that the agent decides what snapshot of the code to read. An agent choosing poorly can lead to it making incorrect deductions. If you know the code well enough (as I did with the memory leak bug), you can successfully push back.

But increasingly, as we transition to a world where all code is written by agents, software engineers’ familiarity with the code is going to plummet. This is borne out in this post about the “latency bug”, where Claude’s initial incorrect analysis seemed plausible enough to me. Despite reading every single one of the 9000 lines, I still didn’t have sufficient expertise in the code to push back against Claude’s initial incorrect deduction. Relying on human pushback to prevent an agent from going off the rails during a debugging session is simply not going to scale.

The other reason I call code an unreliable narrator is because the state space offered by the code is simply too large. Successful debugging hinges on whittling down the state space until you’re left looking at a small nucleus of code that’s responsible for the bug. There are far more effective ways for an agent to do this than to start by reading the code. The next section covers them.

Agentic Debugging Is a Constraint Satisfaction Problem

Treating debugging as an open-ended problem where an agent is free to conjure any hypothesis under the sun after reading some cherry-picked part of the code is tantamount to giving an agent enough rope to hang itself.

Debugging works best when it’s treated as a constraint satisfaction problem. The constraints help prune the state space. More constraints give the agent a better chance at winnowing down spurious theories early.

Without constraints, an agent is free to present any one of the myriad hypotheses as the root cause. However, when constraints are at play, the agent has a different job. It can no longer pick any one plausible hypothesis out of many and pass that off as fact; rather, it now needs to find the hypothesis that is fully consistent with every constraint on the table.

A Two-Tier Framework For Deriving And Harnessing Constraints

It helps to visualize a two-tier framework for identifying and choosing the best constraints.

Bottom Factual Tier

The bottom tier is the factual tier. It consists of hard facts and data gathered from a variety of sources. We should aim to be as open-minded as possible and collect data from every possible source available to us, such as error logs, crash reports, stack traces, profiler data, telemetry data, system traces, kernel data, DNS data, blackbox measurements, performance counters, user reports, service configurations, feature flag values, dependency versions, deployment configurations, version control history, platform configurations, sidecar configurations, etc.

These facts are not conjured by the agent but gathered with the help of external sources or automated tooling. All an agent can do is observe or analyze this data, or request additional data. If the agent has doubts about how the data is gathered in the first place, it can register its concerns with a user, but otherwise will not tinker with the facts. The facts are immutable, and not something an agent can alter or dispute.

Top Derivative Tier

The top tier is the derivative tier. It comprises everything an agent can infer from the factual tier, such as hypotheses, conclusions, conjectures, suspicions, narratives, theories, and so forth. I’ll be referring to these solely as “hypotheses” going forward.

The derivative tier is the one where an agent sounds most convincing and also where it’s most often very wrong, primarily due to misinterpreting or ignoring the facts, and as such is the one most deserving of an outsized share of scrutiny and scepticism. A hypothesis is only as strong as the facts underpinning it, and agents often demonstrate a strong propensity for drifting away from facts while building hypotheses.

In my experience, agents excel at debugging only when the derivative tier is strongly supported by the underlying factual tier. Agents flounder when the derivative tier is divorced from the factual tier and when any hypothesis proffered is untethered from facts.

How do the two tiers relate to constraints?

I treat every fact in the bottom factual tier as a constraint.

A hypothesis from the derivative tier is only valid if it satisfies all constraints. When a hypothesis doesn’t satisfy all the constraints, it should be discarded.

Diagnosing a bug now becomes a problem of finding the hypothesis in the top tier that satisfies every fact from the bottom tier.

Single vs Multi-Cause Hypothesis

In most cases, a bug has a single root cause. In the case of the latency bug, for example, there was a single cause (the new codepath not using the sharded client) that explained every fact. The bug only hurt at production scale, but scale was the condition under which the cause showed up, not a second cause.

In some cases, there might be two or more contributing causes behind a bug. In the case of the memory leak, for example, two causes had to take effect in conjunction for the bug to manifest.

The first cause was that a recent change had added a process-wide cache that kept an authentication engine alive for the lifetime of a process and reentered it on every retry. The second cause was that the state machine inside the engine had been designed to be single-use only, with an exit condition that assumed it would never be reentered once exhausted. Neither of the two causes alone explained the memory leak, but together they did.

Another example is a metastable failure, in which a “system enters a bad state that persists even after the trigger is removed” (Bronson et al). Consider an incident where we see a brief drop in server capacity that overloads a service. A client with unbounded retries could keep the server in an overloaded long after capacity issue is fixed. There are two “facts” at play here — the failure began right after the capacity drop, and that it persisted after capacity was restored. The “capacity drop” hypothesis explains the first fact but not the second. The “unbounded client retries” hypothesis explains the second but not the first. Together they explain both.

In both the cases we saw above, the combination if hypothesis as a whole must satisfy every fact in the factual tier. Moreover, each cause must be strictly necessary, and removing it from the combination should leave some fact unexplained (remove the retries and the service recovers once capacity returns; remove the capacity drop and the retries never happen to begin with).

This might a seem a tad hard to reason about, but the subsequent sections propose useful guidelines to enforce this principle. Ultimately, the goal should never be to explain away inconvenient facts by coming up with more complicated hypotheses.

How To Use The Two Tiers

The goal isn’t to validate every possible hypothesis, which can be time consuming and wasteful of token usage.

Rather, the goal should be to find the hypothesis that satisfies every fact as early as possible in the debugging process.

I refer to this process of choosing which hypothesis to pursue over the others as steering.

Once an agent has been steered towards the hypothesis most likely to be the cause of the bug, the next thing the agent has to do is validate that hypothesis.

I use a three step process to this end. The first two steps steer the agent, and the third step validates the hypothesis.

1. Step 1: Isolate high-fidelity signals in the factual tier
2. Step 2: Isolate the most promising hypothesis in the derivative tier
3. Step 3: Validate the most promising hypothesis against every constraint in the factual tier

Step 1: Isolate high-fidelity signals in the factual tier

The factual tier has many data points. Not all of them are equally valuable. In fact, some of it might be active noise that doesn’t help an agent to be steered towards most promising hypothesis. The agent first needs to identify the data points in the factual tier that are actually related to the bug. Agents do this best when they are given a high-quality control.

In the case of the “latency bug”, the control was a client running with the feature flag disabled, against a server that also had the feature flag disabled. The variable was the case when the feature flag was enabled on both the client and the server.

I was originally hoping to find a discrepancy in TCP handshake time but couldn’t find one. The control made an unexpected data point stand out instead. The control’s DNS lookup returned around 5 addresses, while the variable’s returned around 400.

If a given data point is seen in the variable (around 400 addresses from the DNS lookup), but not the control (only around 5), it can be considered high fidelity signal. If the same data point shows up in both the control and the variable (the TCP handshake time), it’s just noise that shouldn’t be pursued any further.

Noise can still be useful because it can help eliminate certain hypotheses. For example, an error that shows up in the logs of both the control and the variable can’t be what broke the variable, so any hypothesis built upon that error can be discarded. But noise never tells an agent where to look.

Without a control, an agent can’t tell which of the many promising data points in the sea of facts are signals. Whittling down all available data points to find a small number of high-fidelity signals is the chief purpose of using a control.

How To Pick The Right Control?

For simpler debugging scenarios (like the “latency bug”), I’ve often found that the “happy path”, where everything works correctly, makes a good control for an agent. For trickier bugs, the “happy path” can differ from the variable in so many ways that high-fidelity signals can be hard to spot. A better control in such cases is often a “sad path” that is also failing, but in a different, louder way than the variable. A sad path can also be the cheapest control, because it often already exists in the data you have.

“The hang” is an example of the second point. To get a “happy path” control, I could’ve run the daemon on my local laptop, connected it to the remote storage system, and done reads and writes after auth succeeded. But that meant downloading, setting up, and running a daemon I’d otherwise have had no use for. Instead, I used bug reports from users who had hit legitimate auth failures because they lacked the right permissions. Those failures were loud (they left errors in the logs), whereas the hang left nothing, and this “sad path” was a good enough control to compare the “hang” cases against and make progress.

Ultimately, a control is only as good as your confidence that it isn’t broken in the same way as the variable. Here, the permission failures had a different, well-understood cause, which gave me that confidence.

Step 2: Isolate the most promising hypothesis in the derivative tier

Once we’ve identified high-fidelity signals, an agent should never boil the ocean by validating every possible hypothesis. Nor should an agent just take a wild guess and pick a random hypothesis to validate.

The agent should be fastidious about isolating a small set of highly promising hypotheses that would lead it to the correct root cause.

This, in turn, requires an agent to determine which of the myriad high-fidelity signals are strong ones. Once an agent has determined this, it can automatically know which hypotheses are the most promising ones that it ought to pursue next.

In the case of the “latency bug”, both the server P99 latency and the DNS lookup were high-fidelity signals, as both of them showed up in the variable but not in the control. However, only one of these led to the right “root cause”.

If we treat the server-side P99 latency signal as a constraint, it’s one that’s satisfied by almost any hypothesis: a server-side regression, expensive cert reloading, a client-side routing regression, a server-side capacity issue, event loop stalls, throttling, slow upstream, etc. Any of these scenarios could’ve caused a P99 latency regression seen on the server. Claude’s initial attribution of the latency bug to inefficient cert reloading was convincing precisely for this reason.

A signal is considered strong when very few hypotheses satisfy it. Ideally, the number we’re looking for is 1. When too many hypotheses satisfy a signal, it’s a weak one.

One way to distinguish strong signals from weak ones is by generating a matrix which maps every plausible hypothesis to every high-fidelity signal, and count how many hypotheses each signal is consistent with. The hypotheses-by-fact matrix for the latency bug would look like this.

Reading the table column-by-column makes it easy to spot which signal leads to a hypothesis worth pursuing. The P99 signal fits all seven hypotheses whereas the DNS lookup fits exactly one, and that hypothesis (client-side routing) is the one to validate in Step 3.

As a rule of thumb, I’ve found that service SLOs, SLIs etc often make for weak signals. They’re often the symptoms of the bug but don’t help an agent isolate the hypothesis that can lead it to the root cause.

When Every Hypothesis Falls Short

Strong signals don’t always emerge from comparing the variable against the control. Sometimes none of the existing hypotheses can satisfy all the facts at the same time. This tells an agent that something is wrong with the current set of hypotheses.

The “memory leak” bug had exactly this kind of mismatch hiding in plain sight. The bug report came with three facts:

1. The leak was introduced in a recent version of the code. The bug report came with the version where the regression was introduced as well as the last known good version. The most significant code change between those two versions was the recent amortization work.
2. Amortization wasn’t enabled for the user who filed the bug. It was opt-in only, and the user hadn’t changed the defaults.
3. The heap profile pointed at the state machine’s plain recursive call (runStateMachine on line 562), not at the amortized entry point (dedupedRunStateMachine on line 683)

Claude’s initial hypothesis was that the leak was due to a bug in the state machine and that bounding retries was the solution. If the memory leak was indeed due to a long-standing bug in the state machine, it wouldn’t have manifested only in a recent version.

Claude’s next suggestion was that the bug was due to enabling amortization. But amortization was off by default, so that couldn’t have caused the memory leak either.

Drawing up the hypotheses-by-fact matrix makes this mismatch easy to see:

To see whether any hypothesis fits, we scan the matrix row by row, instead of column by column. A hypothesis is only valid if none of the facts contradict it.

None of the hypotheses satisfied all three facts (no row in this matrix is all ✓), which was a sign that something was wrong with the current crop of hypotheses. Before reaching for a brand new hypothesis, an agent should try some cheaper tactics on for size first.

I’ve found these 5 tactics to be helpful.

1. Reverify how the facts were gathered. A fact gathered from the wrong host, during the wrong time window, or with the wrong version of a tool can make the right hypothesis look wrong. An agent can’t alter the facts, so if a fact looks suspect, the agent should ask whoever gathered it to verify it again.
2. Split hypotheses that are too coarse-grained. For example, suppose the P99 latency of a service regressed after a deploy, and the deploy had two code changes, one that changed a connection pool configuration, and the other that added more verbose logging. “The deploy caused the regression” is the immediate hypothesis that an agent might reach for, but it’s too coarse-grained, as it doesn’t tell the agent exactly what about the deploy caused the latency regression. Instead, if the agent starts with two finer-grained hypotheses (the first being “the new connection pool configuration caused the regression” and the second being “the more verbose logging caused the regression”), the agent now has two hypotheses that are easier to verify individually. A fact that contradicts one of them doesn’t rule out the other.
3. Merge hypotheses that are too fine-grained. On the flip side, an agent can come up with scores of very fine grained hypotheses that each explain one symptom of the bug, but none of which explains all of them. For example, let’s suppose the P99 latency regressed on dozens of hosts at the same time due to network degradation impacting an entire availability zone. An agent might come up with fine grained hypotheses like “host 17’s disk is slow”, “host 42 is seeing GC pauses”, or “host 88 has a noisy neighbor”. Each of these hypotheses can explain why P99 regressed for a single host, but none of these hypotheses explain why all the hosts in the AZ saw a P99 regression at once. Merging these into a coarser grained hypothesis (“something common to all the hosts in that availability zone degraded”) will satisfy every fact and point an agent at some shared entity (like the network, or storage) as the root cause of the degradation.
4. Combine complementary hypotheses. Sometimes we might have two hypotheses that each satisfy some of the facts, but neither satisfies all of them, and neither contradicts any facts. In such cases, an agent might want to combine the hypotheses. Unlike merging, combining doesn’t replace them with one broader hypothesis. Combining keeps both the individual hypotheses, because each explains what the other can’t. For example, think of the case where requests start to fail right after a deploy, but only on hosts running a beta version of the operating system. “The deploy introduced a bug” explains the timing constraint, but not why only some hosts fail. “The beta version of the operating system has a bug” explains why the failure is only seen on some hosts, but not why the failures only started after the deploy. Combined, however, they satisfy every fact. Each hypothesis in the combination is still necessary. Remove either hypothesis, and some fact is left unexplained.
5. If tactics 1 to 4 fail, an agent can formulate an entirely new hypothesis, but it should be one guided only by the existing facts. If no more hypotheses can be generated based on the available facts, the next step is to gather more facts via Progressive Signal Discovery, described in the next section.

In the case of the memory leak bug, the right hypothesis could’ve been discovered working through the 5 tactics detailed above.

Tactic 1 (reverifying how the facts were gathered) wouldn’t have helped, because all three facts held up.

Tactic 2 (splitting a coarse-grained hypothesis), however, would’ve certainly helped.

Claude’s “amortization work caused the memory leak” hypothesis bundled two separate ideas (“process-wide cache, enabled by default” and “the amortization, disabled by default”) something Claude itself had spotted while retracting the claim after my pushback. Had Claude split the “amortization work caused the memory leak” hypothesis into finer grained hypotheses instead of brushing it aside wholesale and going round the houses proposing four additional hypotheses, it would’ve been one step closer to the root cause.

Splitting the hypothesis into two alone, however, wouldn’t have been enough. The “process-wide cache” would’ve explained why the leak began in a recent version and why it happened even when amortization was off, but it wouldn’t have explained why the heap profile pointed at the plain recursion. The “single-use state machine” that piled up a frame on each re-entry would’ve explained the heap profile, but wouldn’t have explained why the leak only started recently. This is where tactic 4 (combining complementary hypotheses) would’ve come in. Together, when these two hypotheses were combined, they satisfied every fact (this is the multi-cause hypothesis we saw earlier). Each of the two hypotheses was necessary. Remove either one of them, and some fact is left unexplained.

For the memory leak bug, after taking the initial mismatch into account, and after the subsequent application of tactics 2 and 4, the hypotheses-by-fact matrix would’ve looked like the following:

We finally have a row in this matrix that is all ✓, and that’s the most promising hypothesis, ready for validation in Step 3.

Asking an agent to generate this matrix is a cheap and repeatable way to apply the framework to narrow down the number of hypotheses until only the most likely one remains. It turns both questions (“how many hypotheses satisfy each signal” and “whether every hypothesis falls short”) into something that can be easily inspected.

Progressive Signal Discovery

If every high-fidelity signal from Step 1 turns out to be weak (satisfied by too many hypotheses) or every hypothesis still falls short even after applying “the five tactics”, the next step isn’t to keep generating more hypotheses.

Instead, the agent should go back to Step 1 and gather more facts from a different layer or view of the system. I think of this as progressive signal discovery.

When gathering facts for the factual tier, an agent should start with the the lowest hanging fruit, data that’s already available (data that came as a part of the bug report) or data that can be gathered very cheaply (such as logs, telemetry, and configuration).

If this data produces no strong signal, the next step should be to gather more data from “one layer down”, and repeat this process until a strong signal emerges. Each layer narrows what the next layer has to explain.

We should stop descending as soon as a strong signal emerges. The goal is to find the strongest signal, not the deepest one.

The “latency bug” followed this pattern, where it took gathering data from two layers:

Layer 1: agent looked at telemetry, configuration, code, and previous postmortems, which produced a high-fidelity signal (server-side P99 regression) that was weak

Layer 2: required an agent to ssh into a live production host to gather the high-fidelity signal (DNS lookup) that was strong and led to the right hypothesis

“The hang” took gathering data from four layers until a strong signal was found

Layer 1: the logs (completely uninformative)

Layer 2: packet capture output (showed a healthy connection that completed the TCP handshake and then went silent)

Layer 3: process state (showed nothing actively running in the authentication path, which meant something was waiting rather than stuck in a loop)

Layer 4: kernel IPC table (held the strong signal that showed the authentication daemon was still holding the reply right for our daemon’s request)

This is also where agents especially shine. Gathering and analyzing data from each layer requires different tools, processes, and expertise. Humans typically stop after a layer or two. An agent can keep descending, tirelessly, until it finds a strong signal.

Step 3: Validate the most promising hypothesis against every constraint in the factual tier

Once we’ve identified a strong signal, we can use it as a constraint to determine which hypothesis satisfies it. The most promising hypothesis is still only a candidate. To prove that this hypothesis is indeed the root cause of the bug, we need to validate it against every constraint in the factual tier, including the noise.

In the case of the “latency bug”, the client-side routing hypothesis satisfied every constraint in the factual tier, including both the high-fidelity signals and the noise:

• high-fidelity signals: the DNS lookup, the server P99 regression
• noise: the identical TCP handshake times, and every other available data point

If any fact contradicts a hypothesis, the hypothesis should be discarded. When too few facts are available, several hypotheses survive, and the agent has no way to tell them apart.

The “memory leak” bug proves this. Too few constraints were applied to Claude’s initial (incorrect) reasoning. The derivative tier kept growing as Claude kept coming up with more and more hypotheses. What pruned the derivative tier was imposing the constraints already in the bug report: the leak began in a recent version, it happened with amortization off, the heap profile pointed at the plain recursion, and it only happened when no credentials were supplied up front. Only one hypothesis satisfied all these constraints, and it was a combination of two causes (a process-wide cache re-entering a state machine designed to be single-use).

Fat on Facts, Thin on Hypotheses

The factual tier contains all the facts about a bug. As agents excel at processing large volumes of data, we should aim for the factual tier to be as fat as possible. This is achieved when the factual tier is teeming with:

• different types of data from different (and ideally, all available) sources
• same type of data from different instances of the same bug (for example, packet capture outputs from different users who ran into the same bug, CPU profiles from two different production hosts, inode counts from different clients, etc)

When we have an abundance of data, Step 1 (isolating high-fidelity signals) becomes a problem of pattern recognition. If the same pattern repeats across multiple instances of the same bug while missing from the control, that’s likely to be a real signal and not merely coincidence.

An abundance of data also helps turn the problem of isolating high-fidelity signals from noise into one of arithmetic (by looking at the number of bytes sent or received, number of files opened, number of DNS lookups made, number of RPC requests sent, time to first byte etc) and and letting the arithmetic subsequently guide the agent’s reasoning. The 5 versus 400 DNS addresses in the latency bug was exactly this kind of arithmetic.

While the factual tier should be as fat as possible, the corresponding derivative tier should be as thin as possible. A fat derivative tier indicates that an agent is toying with too many hypotheses and hasn’t been successful in narrowing the state space. Too many surviving hypotheses are a liability as they prolong the investigation. A fat derivative tier is usually a symptom of two flaws in the factual tier:

• too few constraints were applied, like the “memory leak” bug, where the facts could eliminate only two of Claude’s six hypotheses and the other four survived by default
• low-quality constraints that are satisfied by too many hypotheses, like the “latency bug” where the server P99 latency was a high-fidelity signal but one that was satisfied by all 7 hypotheses instead of just one

Agentic Repros Require Additional Scrutiny

This might seem counterintuitive to many, but any repro an agent manages to produce should be treated with a healthy degree of scepticism, especially when the agent uses some kind of synthetic harness in the reproduction attempt, or if the reproduction happens in a non-production environment.

At any given time, there might be numerous erroneous, anomalous, or edge case behaviors lurking in the code. Agents can fixate on super rare edge cases that almost never happen, and almost certainly are not the cause of the bug under investigation. An agent can easily reproduce a very obscure edge case, attribute the bug to that, apply a fix, and call it done, while the bug the real user encountered lingers on.

In the two-tier framework proposed above, reproduction belongs in the derivative tier. A repro is a hypothesis, and like any other hypothesis, has to be validated against the factual tier first. A repro proves that a bug exists, but not that it is the bug under investigation. The factual tier, on the other hand, records what actually happened to the user who reported the bug, so facts in there should take precedence over even a repro.

An agent should still absolutely attempt to reproduce a bug, but when it does so, it should pick the strongest high-fidelity signal from the factual tier first, and verify that the repro also produces the same signal. If this cannot be established, the repro might be real, but it’s not one for the bug under investigation. If a repro is falsified, it’s just noise and should play no further part in the investigation.

I speak from experience here, because this is precisely what happened with “the hang”. While I was ploughing through kernel data, a coworker of mine claimed they’d reproduced “the hang”. In their reproduction, an authentication token was sent from the daemon to the remote server in their test harness. The remote server saw the token as expired and rejected it. This can happen if there’s significant clock skew between the client and the server (which was something that was simulated in the test harness used in my coworker’s reproduction). The client saw nothing wrong with the token, so it kept retrying session establishment, which the server kept rejecting. In this scenario the daemon “hung” leading to timeouts at the virtual file system level. The symptoms looked exactly like what the users were reporting.

But the “signals” didn’t match.

There were two signals of interest in this case. The first signal that I cross-referenced was the packet capture output. On every real user reported hang, each connection showed the same packet sequence. In my coworker’s reproduction, the packet capture output started out the same but then diverged significantly from what was seen in the user reported hangs. The second signal was in the kernel’s IPC tables. On every user reported hang, the authentication daemon was still holding the one-time reply right for our daemon’s request. In my coworker’s reproduction, there was no unanswered reply right found in the IPC tables.

Clock skew is a real problem that affects distributed systems, but it wasn’t the problem in this case.

On the other side of the coin is the the “memory leak”. Here the “signal” was a single frame in the heaptrack output. The repro that came as a part of the bug report from the user also produced the same OOM with the same frame in the heap profile. That is an illustrative example where a repro was proven to be trustworthy, because the signal (the frame seen in the heap profile) matched what the user reported.

Pitfalls of “Adversarial” Reviews

Adversarial reviews are often touted as the panacea for all agentic ills, but in my experience, they’re a much weaker safeguard than made out to be, and can fail in predictable ways.

A fundamental problem with adversarial reviews is that they cannot in any way vouch for the completeness of the factual tier. At best, an adversarial reviewer can verify whether a hypothesis satisfies the constraints already in the factual tier. It cannot know when a constraint is absent, so it cannot detect the “too few constraints” problem.

Another problem with adversarial reviews is they run the risk of confirming the main agent’s priors, if the adversarial reviewer uses the same model, has the same context, looks at the same code, pores over the telemetry data, and so forth as the main agent. In such cases, two agents concurring can be seen as one opinion counted twice.

This isn’t to suggest adversarial reviews are entirely pointless. Only that an adversarial review cannot be trusted to work as instrument of verification. It works better as an instrument of elimination, when it tries to contradict, rather than corroborate a hypothesis.

Contradiction is a stronger and easier test because of how the framework operates. A hypothesis is only valid if it satisfies every fact. No number of confirmations can prove a hypothesis, because there there’s always a possibility that there’s some crucial fact missing in the factual tier that an adversarial reviewer is unaware of (the “unknown unknowns”). In the case of the “latency bug”, one of my initial instructions to Claude was to have an independent subagent validate any theories it came up with. The cert reloading theory survived an independent subagent’s validation. The subagent had no way to disprove it, because the fact that disproved it (a trace showing the certificate being loaded only once) hadn’t been gathered yet. Once that fact was available to the factual tier, the hypothesis was invalidated immediately.

How to Incorporate an Adversarial Review Into The Three Step Debugging Loop

Adversarial reviews should be incorporated into every step of the debugging process described above. At every step, the adversarial reviewer’s core objective is the same: try to eliminate something rather than prove its validity.

Step 1: Isolate high-fidelity signals in the factual tier

The adversarial reviewer should first check if the high-fidelity signals also show up in the control. Those that do are noise. They can’t steer the agent.

Next, the adversarial reviewer should check if the control isn’t broken in the same way as the variable, because a control with the same bug turns every real signal into noise.

Finally, the adversarial reviewer should also check how each fact was gathered. A fact gathered from the wrong host, during the wrong time window, or with the wrong version of a tool can make the right hypothesis look incorrect.

Step 2: Isolate the most promising hypothesis in the derivative tier

Firstly, the adversarial reviewer should estimate how many hypotheses satisfy each signal, without pursuing any of them. A signal that almost any hypothesis can satisfy isn’t a strong one, and any hypothesis derived from it is very unlikely to be the root cause.

Secondly, the adversarial reviewer should read the “hypotheses-by-facts” matrix row by row. If none of the hypotheses satisfy all the facts, the adversarial reviewer should verify that the agent tried using tactics 2 to 4 to either merge, split, or combine existing hypotheses into forms that’d make them more amenable to steering the agent before formulating an entirely new hypothesis.

Finally, the adversarial reviewer should check that the agent only descends a layer when every signal is weak, or when every hypothesis still falls short even after applying “the five tactics”.

Step 3: Validate the most promising hypothesis against every constraint in the factual tier

A hypothesis is only valid if it satisfies every fact, so it’s enough for an adversarial reviewer to find a single fact that’s not satisfied by the hypothesis. Requiring citations is a practical way to achieve this. Citations require an agent to fish out evidence from the factual tier. When I was debugging “the hang”, about a third of the claims I challenged could not be backed up by citations and were culled forthwith.

In the case of a combined hypothesis, a contradicting fact should be enough to immediately spell out which individual hypothesis fails. The adversarial reviewer should also try removing each cause from the combined hypothesis to test whether all the causes are strictly necessary.

Putting It All Together

Retooling the Agentic Factory Floor For Autonomy And Accountability

The guiding principle when it comes to building “software factories” should be to have an agent-native debugging process in place where no human involvement will be necessary to diagnose and validate all types of bugs.

Mind The Gap

Every human intervention spells a gap in the process. All the three bugs chronicled in this post required manual intervention for the agent to get to the root cause. Each intervention brought something new to the table (a control, a constraint, a new check) that an agent ought to have procured itself.

That suggests that every human intervention is an opportunity for improvement, and a true agent-native software factory should treat it as such. The number of human interventions per bug is the most honest measure of how “agent-native” a factory really is, and the goal is for this number to be close to zero.

Agents Versus Humans: A Framework for Tradeoffs

Most existing debugging tools, processes, and frameworks were built for a human audience. We don’t default to emitting logs at trace level so as to not drown a human in noise, we aggregate telemetry into dashboards so as to make it easier for humans to interpret, we use visualizations like waterfall diagrams, bar charts, line graphs etc to aid with human interpretability.

A human wants to deal with less data. All abstractions we currently have (aggregation, visualizations etc) are designed to help humans deal with less data. An agent, however, needs more raw data; its strength lies in processing huge volumes of data at a rapid clip. Aggregation — the thing that makes a dashboard useful to a human — is often the thing that destroys the value of the signal for an agent. The server-side P99 latency measurement in the “latency bug” was a perfect metric for a human audience. As a constraint for an agent, it was nearly useless, because almost any hypothesis could explain it.

When the two goals are at cross-purposes, a useful way to reconcile the tension is to:

• optimize the factual tier for agents
• optimize the decision points for humans
• then turn those decision points into a future policy playbook

Agents require a “fat factual tier” to be scientific when it comes to root-causing bugs. It’s the job of the software factory to make sure every bit of data that’s possibly procurable is made available to the agent. This requires the software factory to have access to a large number of data sources: telemetry systems, production environments, source control systems, and ideally even (within reason) end-users’ desktops, devices etc. This poses serious security and privacy concerns, which can hamstring how truly autonomous a debugging agent can be. The software factory needs to devise a mechanism to get an agent graduated access to data sources which are read-only by default, scoped to the current investigation, time-limited, load and rate limit aware, privacy-scrubbed, and auditable.

“Optimizing the decision points for humans” doesn’t mean requiring a human to be responsible for every decision an agent makes. Our overarching goal is still to have minimal to no human involvement required in the diagnosis or validation of a bug. But acting on a diagnosis can involve decisions that carry political risk and accountability, and cannot be solely owned by an agent. Figuring out whether to roll back a new feature that’s slow but otherwise functional, whether to escalate a bug with a different team, whether to drop support for a specific feature, and so forth are examples of such decisions that need human involvement. Such decisions should be codified into future policy that an agent can apply.

Some decisions, however, can’t be made by an agent by simply applying a predetermined, broad brush policy. The “memory leak” had more than one potential fix (turn off the process-wide cache OR make the state machine safe to use inside the process-wide cache), each with different tradeoffs that required a human to decide how best to proceed. The agent’s job in such cases is to present all available choices to the human in an unambiguous way, along with its reasoning.

The Paradigm Shift In Thinking Required In The Agent-Native Vanguard

Becoming truly agent-native requires a few major shifts in thinking, and each of them inverts habits formed in the era when debugging was the sole preserve of humans.

Designing Verification > Understanding Code

Heretofore, a human’s value in the investigative process came from their mental model of the code. As I argued earlier, that mental model is slowly eroding. What ought to replace it isn’t a better mental model of the code (inasmuch as it isn’t going to be possible for engineers to build crystal clear mental models of frenetically changing code going forward) but a better methodology for reliably and repeatably reasoning about outcomes.

Across the three bugs chronicled in this post, my most valuable contributions weren’t knowledge of the code. They were picking a control, choosing a constraint, and rejecting a hypothesis that hadn’t been validated. The engineers who thrive in an agent-native world will be the ones who are good at designing such verification loops, whether or not they understand the code it verifies.

Consequently, designing good verification loops is going to require a more holistic understanding of the systems under verification than simply the code. To have a verification loop that ensures sharding isn’t broken, you’d first need to understand how sharding is implemented, what a regression would look like, and how this can be identified using signals that can be quantitatively measured and compared.

These decisions become a pivotal part of the initial planning process before the coding agent even generates a single line of code. The coding agent will have to determine how critical pieces of data from the code that validate its correctness can be made available to external agents in the most ergonomic manner. Ideally, we want the strongest signals available to debugging agents from the topmost, shallowest layer (data that’s collected by default), rather than having to resort to Progressive Signal Discovery and descend many layers down the stack to find the signal most likely to lead to the right root-cause. In the case of the latency bug, had the application layer logged the number of DNS lookups it was doing, the strongest signal would’ve been available directly from Layer 1, without needing the agent to descend down to Layer 2 to procure this signal via DNS lookup details from a live production host.

Data Collection > Code Comprehension

While we’re on the topic of code, much of today’s agentic tooling features (better indexing, better retrieval, larger context windows, etc) are aimed at helping agents understand code. But code is an unreliable narrator, and in the case of both the “latency bug” and the “memory leak”, the agent’s first wrong answer came from reading the code.

The accurate root causes were derived from facts that an agent typically doesn’t check (DNS lookups on a production host, a heap profile, the kernel’s IPC tables). The software factory’s goal shouldn’t be to make code more readable for agents, but to help the agents gather better facts from the right layer at the right moment, and to do this enough times until the agent has what it needs to verify the right hypothesis.

Default Data Collection > On Demand Data Collection

Typically, the data needed for debugging is collected on an on-demand basis. This prolongs the debugging loop. Ideally, a software factory will have some mechanism to collect as much data as possible by default, so an agent has all the data it needs available to it right off the bat.

This requires the software factory to resort to techniques like having “always on” advanced data gathering from a small subset of production hosts, to perhaps even having a small subset of the production environment reserved purely for agentic experimentation.

Humans as Policy Setters > Humans as Meat Proxies

A large number of developers have devolved into being mindless meat proxies in the past year, where their sole contribution is blindly copying and pasting agent generated text around, or running specific commands on an agent’s behalf. This is the worst use of both the humans’ and the agents’ time. Humans are poor at processing large volumes of text, and agents flounder when having to work through a keyhole.

An agent-native factory’s goal should be to eliminate the use of a human as a meat proxy. Ideally, any human involvement should only be required when there’s some ambiguity involved in the final decision making process of the agent. Such events should further always act as an opportunity for evolving policies the agent can subsequently rely on.

Turn “Skill Issue” Into An “Infrastructure Issue”

Axioms such as “code doesn’t need to be read or understood anymore”, “loops solve everything”, “you just need more context engineering”, “just include more adversarial reviews” abound. Often, even the mildest scepticism of these supposedly self-evident truths get dismissed as a “skill issue”, where the “skill” in question is the ability to conjure rock-solid, bug-free, production-grade software merely by using a handful of “skills”, or prompting an agent effectively.

Conveniently, these dismissals rarely, if ever, go into precisely what the missing “skill issues” are, beyond mouthing the painfully insipid platitude of “just ask Claude to …”, where the “to” happens to be an ever-expanding list of software development practices, tips and tricks distilled into a few paragraphs of a markdown file.

In each of the three bugs chronicled in this post, the agent was more than capable of getting to the root cause by itself, but it didn’t do so. It didn’t set up a control in the “latency bug” until I asked it to. It didn’t reproduce the “memory leak”, or constrain its hypotheses by the facts in the bug report, until I pointed it out. It didn’t back its claims in “the hang” with citations until I demanded them. Each time, what held the agent back wasn’t what it was capable of, but how far it was willing to go.

If an agent only debugs surgically and methodically when the person driving it knows how to get it to do so, then debugging really is a skill issue. The outcome of agentic debugging is then indeed a function of the individual skill of the developer performing the debugging, with the lessons learnt from each investigation staying confined to one person.

A more robust foundation can be built by investing in processes that allow an agent to go the whole hog by itself. The next great leap in agentic debugging is far more likely to come from better factories than relying on individuals to prompt better.

What Software Factories Are Missing

Almost all of the software factory implementations I’ve seen so far stress the importance of verification but gloss over the specifics of how the pre-production verification ought to happen. There’s also a lot of chatter about leveraging techniques like property based testing, formal methods of verification etc as a part of the agentic verification loop. What’s missing from these discussions is any mention of incorporating the kind of signals described in this post which are far more exacting at validating a change than just looking for passing tests.

In a book I published back in 2018, I’d proposed this testing quadriptych which details all of testing options available at each phase of the software development lifecycle. Agents are far better equipped than humans to conduct a suite of disparate tests at each of these phase, cross-referencing the data between tests, spotting regressions and anomalies, and more.

Develop Against Production

Once we have the infrastructure in place to allow agents to perform a panoply of such tests, we can shift testing left by bringing production closer to development. If you, like me, believe that all software development is going to happen via cloud agents, it’s imperative to ensure that agents can verify their changes as early as possible in the development lifecycle, and not wait until much later in the process to detect problems. This requires the development process to happen against an isolated slice of the production environment, with read only access to all signals, so a change can be tested against real production behavior as soon as it’s generated.

“Full Stack In A Box” Is Still An Anti-Pattern

One bad trend I’ve seen make a comeback is what I call the “full stack in a box” approach. About a decade ago, I wrote a post explaining why the “full stack in a box” approach for testing microservices was a bad idea. It’s an even worse idea to use this approach with agentic software development.

“Full stack in a box”, for those who’re not familiar with the term, is the approach where you run your entire production stack locally, including databases, caches, queues, etc, with agents monitoring or debugging some component of the “full stack”.

The main reason I believe “full stack in a box” is a bad approach for agents to use for debugging and verification is because the factual tier, which is the very linchpin of the agentic debugging loop described in this post, cannot be accurately replicated locally to match production. When your factual tier is unreliable or sparse, it’s very difficult for an agent to faithfully verify the change or zap through to the root cause.

It’s much more effective to ensure your debugging agent has the all the facts it requires directly from the production environment where the bug is observed, rather than spin up elaborate local test harnesses. Not to put too fine a point on it, but yes, this means figuring out how to give an agent access to your production environment so it can gather all the data it needs securely.

If it’s of any consolation, the security architecture underpinning this can be repurposed for many adjacent use cases, like running integration testing in production, so getting it right once is more than worth the while. Critically, this also enables development agents, and not just debugging ones, to tap into the same signals which allow for early regression detection.

Repurpose Debugging Framework For Integration Tests And Deploy Gating

The framework proposed in this post for debugging — picking a good control, identifying high-fidelity signals, with an adversarial reviewer involved in each stage — is something that would also work very well for integration tests, end-to-end tests, pre-deploy gates, automated canary analysis, and so forth.

With agents, the goal of testing isn’t just observing a binary final outcome (success vs failure), but checking for parity when it comes to every high-fidelity signal. Many of the signals used in past debugging incidents can also be baked into pre-deploy gating checks, to safeguard against regressions. In the case of the “latency bug”, if the pre-production testing environment had checked that the client’s DNS lookups matched those on the previous successful run, the routing bug would’ve been caught before it reached production, even at pre-production’s small scale.

Conclusion

The focal point of many public discussions around agentic coding happens to be the ultimate potential of these tools. The discourse seldom captures real world practicalities, or the realities of people building production software who have critical business needs and serve real users.

However, what I’m seeing happen in reality (and I’m willing to wager that mine is by far not an isolated experience) is that even as models become more capable, the engineers using them aren’t necessarily becoming vastly improved versions of themselves. In fact, agents can often amplify engineering weaknesses, both individual as well as organizational.

Engineers who struggled with debugging complex problems continue to do so even with access to Claude Code, but often with far worse ramifications. While prior to using coding agents they may have sought help from a coworker when they failed to debug a problem, now I see them open PRs generated by Claude’s complete misdiagnosis of a problem. At best, such “non-fixes” fail to fix the problem. At worst, they introduce new bugs or break some other implicit functionality with a wide blast radius.

Codebases that accrue a plethora of such “non-fixes” become increasingly riddled with incongruence. A large number of “readability” complaints come from people manoeuvring such codebases that have degenerated into bloated artifacts of incoherence after being fed PR after PR of utterly empty calories.

Likewise, engineers who didn’t have much of an eye for detail aren’t suddenly producing code that’s well-structured, well-layered, mindful of performance and security implications, attentive to the impact on dependents (both direct and transitive), diligent about realistic edge-case handling, meticulous about telemetry soundness, etc. Sure, you can have a “skill” for all of this, but agents can take the “empty calories” approach, where each of these requirements are met cosmetically but not substantively.

The embrace of agentic paradigms for software development comes with trading understanding for speed. This doesn’t necessarily have to spell doom and gloom. It’s possible to ship high quality software at breakneck speed without sacrificing reliability, but it requires us to stop treating agents as oracles. Replacing a deep understanding of code with a deeper, higher-level understanding of systems, and a rigorous, regimented verification process like the one described in this post is the path forward.

This process, when applied consistently, can help build enough confidence in an agent’s ability to debug gnarly issues without requiring a human in the loop. While it calls for great focus, creativity, and assiduity to build such processes (and far be it from me to stake a claim to familiarity, let alone mastery in this area), the payoff is often a great reduction in time and effort required to debug any problem that might pop up in any layer of the stack you’d otherwise be too daunted to explore.

We now have the tools at our disposal that allow us to dream bigger and be inordinately more ambitious with our bets. This also requires a change in the epistemic discipline engineers use when working with agents. With a degree of creativity, and a sense of adventure, these ambitious bets are within easy reach, if we can only rise to the challenge.