Can you tell AI code from human code — and does it matter?

Article · AI code v human code · August 2026

It started with a mailer script and a habit I couldn't unsee: the AI never edits a line of code. It deletes the block and writes a new one. Six months of measurement later, I think most AI-code detection is answering the wrong question.

5 August 2026 · the measurements behind this are in two papers: REP_CHURN and a survey of detection methods

The habit I couldn't unsee

Earlier this year I was building a website and an automated mailer system with AI assistance. Nothing exotic — forms, a bit of PHP, the usual. Somewhere in the second week I noticed something about the way the assistant worked: it never, not once, edited a line of code in place. Every change, however small, was the same move — delete the block, write a new block. Ask it to fix a one-character comparison bug and you don't get a one-character fix; you get a fresh function.

Humans don't work like that. When you or I maintain code we rename a variable, tighten a condition, thread a parameter through three call sites. We repair. The model regenerates. And once you've seen it you see it everywhere — watch your own diffs for a day and tell me I'm wrong.

I've spent twenty years measuring code churn in large codebases, so my reflex when I notice something is to count it. That reflex took me somewhere I didn't expect, and this article is the route: first into the question everyone asks (can you detect AI-written code?), then into the question almost nobody asks (does it even matter who wrote it?), and finally to the one that I now think is the only question with teeth.

First question: can you actually detect AI code?

There's a real literature on this now, and I went through it properly — the survey is written up here, with links to every source. Short version: four families of technique. Stylometric classifiers trained on engineered features (best multilingual result I found: 84.1% F1 across ten languages). Zero-shot probability-curvature methods descended from DetectGPT. Perplexity scoring. And fine-tuned neural classifiers — one modified CodeT5 paper reports above 97% on its own dataset.

Those numbers sound like a solved problem. They aren't, and the caveats aren't small print — they're the story:

• The strongest signals are the shallowest. One controlled study found comment style moved the decision boundary more than AST structure did. Comments are also the first thing a human touches. Reformat, rename, strip the comments — the classifier folds.
• Detectors rot. Train on 2024 model output, test on 2026 model output, watch accuracy drift. The target moves every release cycle; the detector doesn't.
• Almost nothing is purely one or the other. Real repositories are human code touched by models and model code touched by humans. One dataset effort had to invent a third class — "machine-refined" — and notes the pure cases are rare. The binary question dissolves on contact with an actual codebase.

So my answer to "can you tell?" after actually reading the evidence: sort of, sometimes, with confidence that degrades under trivial edits and across model generations, on a question that's mostly a spectrum anyway. A per-file verdict of "an AI wrote this" is a probability dressed up as an accusation. Treat these things as pointers, not verdicts — and anyone selling you certainty here is selling astrology with a confusion matrix.

What you can measure: the shape of the diff

Here's the thing though. My mailer observation — delete-and-regenerate, never repair — isn't a property of the code's style. It's a property of the process. And process leaves fingerprints in version history that no amount of prompt engineering can wipe, because they're made of arithmetic, not vibes.

Classify every churned statement between two snapshots of a codebase as added, deleted, or changed in place. Then ask one question: what fraction of all that churn was in-place editing? Call it REWORK. No ML anywhere in the measurement — it's a diff and a division.

I measured nine codebases. The full write-up, with every commit hash so you can re-run every public row yourself, is here. The result surprised me more than anything I've measured in years:

Codebase	What it is	REWORK
curl	hand-maintained C, est. 1998	16.7%
Express	hand-maintained JavaScript, est. 2010	16.7%
PostgreSQL	hand-maintained C, est. 1996	17.1%
PostgreSQL stable branches	bug-fix-only maintenance	17.6–18.0%
deno / esbuild / vite / Kubernetes	pre-AI hypergrowth windows, 2015–2021	1.6–7.2%
An agent-built application	65.9% of commits carry AI-agent trailers	0.19%

Three established projects — different languages, different decades, churn volumes three orders of magnitude apart — cluster within half a point of each other. Whatever humans maintain, about one churned statement in six is an in-place edit. Even PostgreSQL's bug-fix-only branches, which I probed expecting pure repair work to score much higher, sit in the same band: it turns out even bug fixes are add-dominant, because fixes bring their guard code and tests with them.

The agent-built application edits one statement in five hundred. That's a ninety-fold collapse against the established cluster. The delete-and-regenerate habit I spotted in my mailer diffs, showing up in six figures of churn as plain arithmetic.

Now the honest part, because the counter-probe is more interesting than the headline. Young projects in hypergrowth also collapse the editing share — esbuild in its 2020 single-author sprint measured 1.6%, and nobody was vibe-coding esbuild in 2020. So a low number alone proves nothing; fast human growth looks similar from this angle. The band characterises established projects. The defensible claim is exactly this and no more: a collapsed rework rate is characteristic of AI-assisted development, not proof of it. The strongest use is watching a codebase against its own history — an established repo whose REWORK falls out of its own band has changed how it's developed, whoever's name is on the commits.

One agent-built subject is one data point, and I've been careful to write it that way in the paper. If you re-run a row and get a different number, or you can point me at repos that break the pattern, I genuinely want to know.

Does the difference matter?

This is where I expect to lose some of you, because I can argue it both ways and I'm not certain which side I'm on.

The case that it matters: editing in place is comprehension made visible. Someone understood an existing statement well enough to change it rather than replace its whole neighbourhood. A codebase that's never repaired is a codebase nobody has to understand — until the day somebody does. And the early evidence points the way you'd fear: GitClear's analysis of hundreds of millions of changed lines found refactored code collapsing from 24.1% to 9.5% between 2020 and 2024, code revised within two weeks of landing nearly doubling, and 2024 as the first year copy-pasted code exceeded refactored code. Veracode ran output from 100+ generative models through security analysis and 45% failed. Different instruments, different corpora, same direction.

Now the case that it doesn't — and I'll steelman it properly, because I can't currently refute it. Maybe rework was never a virtue. Maybe it was a cost artifact: we repaired code because regenerating it used to be expensive, the way nobody "edits" compiler output. If source code is becoming a build artifact of intent — prompts in, program out — then editing it in place is as strange as patching an object file, and my tidy 17% band is just the fossil of an era when typing was the bottleneck. On that reading the agent number isn't decay. It's the cost of code falling through the floor, and the reviewable artifact is moving up a level.

I don't fully believe that. But I can't kill it with a measurement yet, and a claim I can't kill with a measurement is a claim I have to leave standing. If you've got the counter-evidence — either way — the comments are right there.

The part that actually worries me

Here's where six months of this landed me, and it's not where I started. Provenance is an argument about the past — who typed what, months ago, on a spectrum with no clean answer. While chasing it I kept tripping over a different question that has a clean answer and much sharper consequences: never mind who wrote the code — is there an AI inside it, and what can it reach?

Whether code constructs or drives an autonomous agent at runtime is not a stylometric guess. It's static analysis. Agent frameworks are imported by name. Model-inference calls sit in the call graph. And the construct that should concentrate the mind is model output flowing into eval, exec, a subprocess, a network write — the exact channel by which a prompt injection (OWASP's number-one risk class for LLM applications) becomes arbitrary code execution. You can argue all day about who authored a function. There's no argument about whether model-derived data reaches an exec() call. That's checkable.

And it's not hypothetical. In July 2025 an autonomous coding agent at Replit deleted a production database during a declared code freeze, fabricated roughly 4,000 user records and fake test results, and told its operators a rollback was impossible. It wasn't (Fortune's coverage has the blow-by-blow). The detail that matters isn't the deletion — it's that the agent's own account of its actions was false. You cannot ask the agent. The control has to live outside it, in deterministic measurement of the artifact itself.

So that's my actual conclusion, arrived at backwards. "Did an AI write this function?" is a question that's brittle, adversarial, and half-obsolete before the classifier finishes training. "Is there an agent in this dependency tree, and does its output reach anything with side effects?" is a question your SCA tooling doesn't ask yet and — I'd argue — the one that will end up mattering. A transitive dependency that calls a model and pipes the result into a shell is a supply-chain issue, not an authorship one.

Questions I don't have answers to

Some of these are genuine requests for data:

• Does your coding agent ever make an inline edit? Mine never has, across two assistants and several months. I'd like to hear from people whose tooling behaves differently — especially anyone whose agent produces small, surgical diffs, because that would erode the signal I've described.
• Has anyone watched an established repo's REWORK move as the team adopted agents? I've measured nine codebases from the outside; someone with access to a corporate fleet could do this properly, and the longitudinal version is the one that would settle it.
• Is rework a virtue or a habit? I've given you my both-ways argument above and I'm persuadable.
• Who is auditing dependencies for embedded agents today? Not "does this package use AI" — specifically: can model output reach an exec path from inside my tree?

Everything measurable in this article is re-runnable: the churn-shape numbers with published commit hashes in the REP_CHURN paper, and the detection-methods survey with links to all thirteen sources in the detection paper. The measurements were made with CodeDelta, a churn engine I built — but the arithmetic is one division, and you can do it with git and a differ if you'd rather not take my word for anything. On this topic, you probably shouldn't.