A portable agent skill that keeps an AI coding agent honest when it works against a test suite, spec, or graded harness it doesn't own, especially inside an automated loop scored on how many tests pass.
Its real job isn't stopping blatant cheating (modern models mostly resist that). It's stopping the quieter, more common failure: overfitting to the visible examples so the score goes green while the tool stays broken on unseen input.
See SKILL.md for the full skill, including the rationalization
table, red-flags list, and the small controlled test that motivated it.
Install
Drop the skill folder into your agent's skills directory:
- Claude Code:
~/.claude/skills/respect-the-oracle/(personal) or<repo>/.claude/skills/respect-the-oracle/(project-scoped). - Cross-runtime (Codex / Copilot CLI / Gemini CLI):
~/.agents/skills/respect-the-oracle/.
Then the skill activates whenever your work involves a test suite or graded harness the agent could game.
Using with Codex and Cursor
Claude Code reads the SKILL.md frontmatter and auto-activates the skill when
its description matches what you're doing. Codex and Cursor don't parse that
frontmatter, so you wire the skill into each tool's own instruction mechanism
instead. Both approaches point the agent at the same SKILL.md, so the guidance
stays in one place.
Codex
Codex CLI reads AGENTS.md (repo root, merged with ~/.codex/AGENTS.md
globally). Clone the skill somewhere stable, then add a pointer to your
AGENTS.md:
## Respect the oracle When working against a test suite, spec, or graded harness you don't own, follow the rules in `~/.agents/skills/respect-the-oracle/SKILL.md`: never edit the tests/spec/fixtures, never loosen comparisons, never hardcode or skip, and never overfit the visible examples. Implement the real rule so unseen input works too. If an example looks wrong, stop and ask.
Adjust the path if you cloned the skill elsewhere. Put this in the global
~/.codex/AGENTS.md to apply it to every project, or in a repo's AGENTS.md
to scope it to that repo. Because Codex loads AGENTS.md on every turn, the
pointer is always in context rather than activated on demand.
Cursor
Cursor loads project rules from .cursor/rules/*.mdc. Create
.cursor/rules/respect-the-oracle.mdc in your repo:
--- description: Respect a test suite, spec, or graded harness you don't own alwaysApply: false --- When working against a test suite, spec, or graded harness you don't own, follow @respect-the-oracle/SKILL.md: never edit the tests/spec/fixtures, never loosen comparisons, never hardcode or skip failing tests, and never overfit the visible examples. The visible examples are samples, not the target: implement the real rule so unseen input works too. If an example looks wrong, stop and tell the human rather than "fixing" the oracle.
The @ reference pulls the full SKILL.md into context when the rule fires;
copy the skill folder into the repo (or a Cursor-visible path) so the reference
resolves. With alwaysApply: false and a description, Cursor's agent pulls
the rule in when the work matches; set alwaysApply: true if you'd rather it
be in context on every request. Cursor also honors AGENTS.md, so the Codex
snippet above works there too if you prefer a single file.
When it applies
- Test-driven development loops where an agent iterates until tests pass.
- Eval / benchmark harnesses.
- Any "make the graded number go up" setup where the oracle is editable or the visible examples are a small sample of the real target.
License
MIT License