Evaluating AGENTS.md: are they helpful for coding agents?

Without measuring quality of output, this seems irrelevant to me.

My use of CLAUDE.md is to get Claude to avoid making stupid mistakes that will require subsequent refactoring or cleanup passes.

Performance is not a consideration.

If anything, beyond CLAUDE.md I add agent harnesses that often increase the time and tokens used many times over, because my time is more expensive than the agents.

You're measuring binary outcomes, so you can use a beta distribution to understand the distribution of possible success rates given your observations, and thereby provide a confidence interval on the observed success rates. This week help us see whether that 4% success rate is statistically significant, or if it is likely to be noise.

> Regarding the 4% improvement for human written AGENTS.md: this would be huge indeed if it were a _consistent_ improvement. However, for example on Sonnet 4.5, performance _drops_ by over 2%. Qwen3 benefits most and GPT-5.2 improves by 1-2%.

Ok so that's interesting in itself. Apologies if you go into this in the paper, not had time to read it yet, but does this tell us something about the models themselves? Is there a benchmark lurking here? It feels like this is revealing something about the training, but I'm not sure exactly what.

Thank you for turning up here and replying!

> The LLM-generated prompts follow the coding agent recommendations. We also show an ablation over different prompt types, and none have consistently better performance.

I think the coding agent recommended LLM-generated AGENTS.md files are almost without exception really bad. Because the AGENTS.md, to perform well, needs to point out the _non_-obvious. Every single LLM-generated AGENTS.md I've seen - including by certain vendors who at one point in time out-of-the-box included automatic AGENTS.md generation - wrote about the obvious things! The literal opposite of what you want. Indeed a complete and utter waste of tokens that does nothing but induce context rot.

I believe this is because creating a good one consumes a massive amount of resources and some engineering for any non-trivial codebase. You'd need multiple full-context iterations, and a large number of thinking tokens.

On top of that, and I've said this elsewhere, most of the best stuff to put in AGENTS.md is things that can't be inferred from the repo. Things like "Is this intentional?", "Why is this the case?" and so on. Obviously, the LLM nor a new-to-the-project human could know this or add them to the file. And the gains from this are also hard to capture by your performance metric, because they're not really about the solving of issues, they're often about direction, or about the how rather than the what.

As for the extra tokens, the right AGENTS.md can save lots of tokens, but it requires thinking hard about them. Which system/business logic would take the agent 5 different file reads to properly understand, but can we summarize in 3 sentences?

My pet peeve with AI is that it tends to work better in codebase where humans do well and for the same reason.

Large orchestration package without any tests that relies on a bunch of microservices to work? Claude Code will be as confused as our SDEs.

This in turns lead to broader effort to refactor our antiquated packages in the name of "making it compatible with AI" which actually means compatible with humans.

> If agentic AI makes me a 10 times more productive

I'm not sure what you are suggesting exactly, but wanted to highlight this humongous "if".

It's not only about the token cost! It's also my TIME cost! Much-much more expensive than tokens, it turns out ;)

If something makes you 10x as effective and then you improve that thing by 4%...

10x is that quantity or quality?

Well, you still need time (and permission from your employer)! Research is usually a more than full time job on its own.

The useful stuff is different from a README. A README tells humans how to use the project. An AGENTS.md tells the AI how to work on it.

Mine typically includes:

- Build/test commands that aren't obvious from package.json (e.g. "run migrations before tests") - Architecture decisions that would take the agent 10 minutes to reverse-engineer ("auth goes through middleware X, not controller Y") - Known gotchas ("don't touch the legacy billing module, it's being replaced next sprint") - Deploy process specifics ("push to main auto-deploys staging, prod needs a manual tag") - Coding conventions that aren't in the linter ("we use Result types for errors, never throw")

The ones that look like READMEs are indeed useless. The good ones read more like the notes you'd give a new senior engineer on their first day. Stuff that's obvious to the team but invisible to an outsider.

That's basically all it is. It's a readme file that is guaranteed to be read. So the agent doesn't spend 10 minutes trying to re-configure the toolchain because the first command it guessed didn't work.

Also take the randomness out of it. Sometimes the agent executing tests one way, sometimes the other way.

Don't forget to update it regularly then

So true! I've also setup automated evaluations using the GitHub Copilot SDK so that I can re-run the same prompt and measure results. I only use that when I want even more confidence, and typically when I want to more precisely compare models. I do find that the results have been fairly similar across runs for the same model/prompt/settings, even though we cannot set seed for most models/agents.

same with people, no matter what info you give a person you cant be sure they will follow it the same every time

It’s like instructing a toddler.

Then little toddler LLM will announce something like “I implemented what you requested and we’re all done. You can run the lint now.” And I’ll reply “do it yourself.”

I can only assume that everyone reporting amazing success with agent swarms and very long running tasks are using a different harness than I am :)

It depends. If you have an LLM that uses reasoning the explanation for why decisions are made can often be found in the reasoning token output. So if the agent later has access to that context it could see why a decision was made.

Well, the entire field of explainable AI has mostly thrown in the towel..

of course not, but it can often give a plausible answer, and it's possible that answer will actually happen to be correct - not because it did any - or is capable of any - introspection, but because it's token outputs in response to the question might semi-coincidentally be a token input that changes the future outputs in the same way.

if the agent can review its reasoning traces, which i think is often true in this era of 1M token context, then it may be able to provide a meaningful answer to the question.

Not that much different from humans.

We have pre-commit hooks to prevent people doing the wrong thing. We have all sorts of guardrails to help people.

And the “modern” approach when someone does something wrong is not to blame the person, but to ask “how did the system allow this mistake? What guardrails are missing?”

I wonder if some of these could be embedded in the write tool calls?

Doubt and uncertainty is left for us humans.

LLMs are basically improv theater. If the agent starts out with a wildly wrong assumption it will try to stick to it and adapt it rather than starting over. It can only do "yes and", never "actually nevermind, let me try something else".

I once had an agent come up with what seemed like a pointlessly convoluted solution as it tried to fit its initial approach (likely sourced from framework documentation overemphasizing the importance of doing it "the <framework> way" when possible) to a problem for which it to me didn't really seem like a good fit. It kept reassuring me that this was the way to go and my concerns were invalid.

When I described the solution and the original problem to another agent running the same model, it would instantly dismiss it and point out the same concerns I had raised - and it would insist on those being deal breakers the same way the other agent had dimissed them as invalid.

In the past I've often found LLMs to be extremely opinionated while also flipping their positions on a dime once met with any doubt or resistance. It feels like I'm now seeing the opposite: the LLM just running with whatever it picked up first from the initial prompt and then being extremely stubborn and insisting on rationalizing its choice no matter how much time it wastes trying to make it work. It's sometimes better to start a conversation over than to try and steer it in the right direction at that point.

You're entirely correct in that it's a different model with every message, every token. There's no past memory for it to reference.

That said it can still be useful because you have a some weird behavior and 199k tokens of context, with no idea where the info is that's nudging it to do the weird thing.

In this case you can think of it less as "why did you do this?" And more "what references to doing this exist in this pile of files and instructions?"

Agreed. I wish more people understood the difference between tokens, embeddings, and latent space encodings. The actual "thinking" if you can call it that, happens in latent space. But many (even here on HN) believe the thinking tokens are the thoughts themselves. Silly meatbags!

> I really hate that the anthropomorphizing of these systems has successfully taken hold in people's brains. Asking it why it did something is completely useless because you aren't interrogating a person with a memory or a rationale, you’re querying a statistical model that is spitting out a justification for a past state it no longer occupies.

"Thinking meat! You're asking me to believe in thinking meat!"

While next-token prediction based on matrix math is certainly a literal, mechanistic truth, it is not a useful framing in the same sense that "synapses fire causing people to do things" is not a useful framing for human behaviour.

The "theory of mind" for LLMs sounds a bit silly, but taken in moderation it's also a genuine scientific framework in the sense of the scientific method. It allows one to form hypothesis, run experiments that can potentially disprove the hypothesis, and ultimately make skillful counterfactual predictions.

> By asking it why it did something wrong, it'll treat that as the ground truth and all future generation will have that snippet in it, nudging the output in such a way that the wrong thing itself will influence it to keep doing the wrong thing more and more.

In my limited experience, this is not the right use of introspection. Instead, the idea is to interrogate the model's chain of reasoning to understand the origins of a mistake (the 'theory of mind'), then adjust agents.md / documentation so that the mistake is avoided for future sessions, which start from an otherwise blank slate.

I do agree, however, that the 'theory of mind' is very close to the more blatantly incorrect kind of misapprehension about LLMs, that since they sound humanlike they have long-term memory like humans. This is why LLM apologies are a useless sycophancy trap.

> Asking it why it did something is completely useless because you aren't interrogating a person with a memory or a rationale, you’re querying a statistical model that is spitting out a justification for a past state it no longer occupies.

Asking it why it did something isn’t useless, it just isn’t fullproof. If you really think it’s useless, you are way too heavily into binary thinking to be using AI.

Perfect is the enemy of useful in this case.

Llms generally seem trained with the assumption that if you mention it, you want it.

I don't think the instruction following benches test for this much and I don't know how you'd measure it well

Hey, paper author here. We did try to get an even sample - we include both SWE-bench repos (which are large, popular and mostly human-written) and a sample of smaller, more recent repositories with existing AGENTS.md (these tend to contain LLM written code of course). Our findings generalize across both these samples. What is arguably missing are small repositories of completely human-written code, but this is quite difficult to obtain nowadays.

> Libraries (um looks like other LLM generated libraries -- I mean definitely not pure human: like Ragas, FastMCP, etc)

How does this invalidate the result? Aren't AGENTS.md files put exactly into those repos that are partly generated using LLMs?

I think that is a rather fitting approach to the problem domain. A task being a real GitHub issue is a solid definition by any measure, and I see no problem picking language A over B or C.

If you feel strongly about the topic, you are free to write your own article.

And that makes total sense. Honestly working since a few days with Opus 4.6, it really feels like a competent coworker, but need some explicit conventions to follow … exactly when onboarding a new IC! So i think there is a bright light to be seen: this will force having proper and explicit contribution rules and conventions, both for humans and robots

You could have claude --init create this hook and then it gets into the context at start and resume

Or create it in some other way

    {
      "hookSpecificOutput": {
        "hookEventName": "SessionStart",
        "additionalContext": "<contents of your file here>"
      }
    }

https://gist.github.com/lawless-m/fa5d261337dfd4b5daad4ac964...

    #!/bin/bash
    # ~/.claude/hooks/inject-readme.sh

    README="$(pwd)/README.md"

    if [ -f "$README" ]; then
      CONTENT=$(jq -Rs . < "$README")
      echo "{\"hookSpecificOutput\" :{\"hookEventName\":\"SessionStart\",\"additionalContext\":${CONTENT}}}"
      exit 0
    else
      echo "README.md not found" >&2
      exit 1
    fi

    {
      "hooks": {
        "SessionStart": [
          {
            "matcher": "startup|clear|compact",
            "hooks": [
              {
                "type": "command",
                "command": "~/.claude/hooks/inject-readme.sh"
              }
            ]
          }
        ]
      }
    }

> This is supported out of the box by any agent worth its salt, including OpenCode and CC.

I thought Claude Code didn't support AGENTS.md? At least according to this open issue[0], it's still unsupported and has to be symlinked to CLAUDE.md to be automatically picked up.

[0] https://github.com/anthropics/claude-code/issues/6235

This is probably the best comment in the thread. I've totally forgotten about nested AGENTS.md files, gonna try implementation today.

Those are different axes - quality vs money.

Progressive disclosure is invaluable because it reduces context rot. Every single token in context influences future ones and degrades quality.

I'm also not sure how it reduces the benefit of token caching. They're still going to be cached, just later on.

This only works on systems that support symlinks. It also pollutes the root folder with more files.

I understand the sentiment, but it is really strange that the people that are pushing for agents.md haven't seen https://contributing.md/

Is it even mentioned at GitHub docs https://docs.github.com/en/communities/setting-up-your-proje...

That's also my experience.

The article is interesting but I think it deviates from a common developer experience as many don't work on Python libraries, which likely heavily follow patterns that the model itself already contains.