TOKENME.txt – Decoupling declarative codebase state from AGENTS.md
github.comAuthor here. Over the past few months of running local multi-agent coding pipelines (where orchestrators hand off tasks across isolated architect, builder, and adversarial auditor modes), we kept hitting the same context bottleneck with AGENTS.md and CLAUDE.md files.
AGENTS.md is great for static, imperative rules: build flags, test commands, lint boundaries, and judgment constraints. However, standard guidelines warn that AGENTS.md starts losing instruction-following compliance once it grows past ~150 lines.
In practice, as a repository scales, developers (or init scripts) inevitably stuff dynamic codebase state into AGENTS.md: directory trees, database schemas, state-flow rules, and exported function catalogs. Mixing imperative rules ("how the agent must behave") with high-churn declarative state ("what currently exists in the repo") causes two failures: 1. Operational guardrails get buried in architectural noise, causing agents to skip verification steps or ignore routing rules. 2. Static context files drift commits behind the working tree, leading stateless coding agents to write duplicate helper functions (e.g., building a brand-new string parser because they don't know one already exists two folders over).
We formalized TOKENME.txt (v1.1) under CC-BY 4.0 to split repository context into a strict three-file triad: - README.md (Narrative): For human onboarding. - AGENTS.md (Imperative): Capped at 60-150 lines for static execution rules, tool permissions, and a top-level lexicographical trade-off hierarchy (Priority A > Priority B when Condition C). - TOKENME.txt (Declarative): A live, auto-synced, raw .txt system state and symbol catalog built strictly for machine ingestion at every handoff.
Why raw .txt and hard sharding thresholds? - Passing Markdown with nested code fences and pipe tables across web UIs, clipboards, and IDE extensions routinely triggers silent parser truncation. Raw .txt guarantees zero syntax collisions. - A monolithic context file still blows up token budgets on large repos. The v1.1 spec enforces a mathematical sharding trigger (>3,000 tokens, >250 lines, >100 exported symbols, or >=2 runtimes), at which point the root TOKENME.txt converts into a <=1,000-token Global Router pointing to scoped `/src/<domain>/TOKENME.txt` shards (cutting our per-handoff token overhead by 74.4%).
The repo includes the full v1.1 spec, the sharding protocol, and a self-referential bootstrap block you can drop into an existing pipeline. Very interested in critical feedback on the sharding thresholds and metadata schema for v1.2.