I Scanned 1M Domains for llms.txt | DialtoneApp

Findings

The numbers changed the article

The first surprise was not that a few AI-native startups publish model instructions. The surprise was how normal this has already become.

Landscape

This is not one industry

The LLM-readable web is already spread across publishers, stores, SaaS companies, developer docs, finance, education, security, travel, healthcare, and local businesses.

Automation

The biggest source is boring, and that is why it matters

A huge part of adoption appears to be CMS and SEO automation rather than deliberate AI strategy.

I found 9,477 domains with generated-file signatures. Of those, 5,192 looked like Yoast SEO output and 2,189 looked like All in One SEO output. That is the most important adoption clue in the whole scan. Standards spread when they become a checkbox in tools people already use. Most website owners will not sit down and write an AI retrieval policy. They will update a plugin and suddenly have one.

The generated files are uneven. Some are just page lists. Some include generic text explaining that the file is used by LLMs to index the site. Some accidentally expose poor titles, boilerplate, or malformed content. I saw gzip-compressed responses saved with .txt names, UTF-16-like text, image bytes, and HTML fallbacks. The exporter had to decompress 711 gzip files before classification. The upstream crawler should preserve content type and content encoding more explicitly because otherwise the corpus lies to you.

But "messy" does not mean "fake." It means the layer is becoming mundane. That is how robots.txt, sitemaps, schema markup, canonical URLs, Open Graph tags, and product feeds all arrived. First the hand-authored examples get attention. Then the tooling normalizes the pattern. Then everyone has a version, including sites that do not know what it does.

That automation explains why the corpus has two personalities. One personality is intentional: a company writes a compact policy, names its canonical docs, and gives agents instructions. The other is accidental: a plugin dumps titles, descriptions, and links into a file because "AI visibility" became another checkbox beside XML sitemaps and Open Graph tags. The accidental version is often less beautiful, but it may be more important because it can scale through the long tail of the web.

A generated file also creates a new maintenance problem. Search pages can be wrong, but people still see them. A bad llms.txt can be wrong invisibly. It can point models at old docs, expose placeholder titles, classify a site poorly, or make every page look equally important. For a site owner, the file should probably become part of release hygiene: when pricing changes, docs move, product pages disappear, or a policy changes, the machine-readable briefing needs to change too.

The generated signatures were also useful for separating strategy from adoption. If a file says "Generated by Yoast SEO" or "Generated by All in One SEO," I treat it as evidence that the ecosystem is adopting the format, not evidence that the business has a mature AI answer strategy. If a file has custom citation rules, regional caveats, tool guidance, or anti-hallucination instructions, I treat it as an intentional artifact. The next crawler pass should split those classes more aggressively.

Control Plane

llms.txt is becoming more than an index

The strongest files do not just list URLs. They tell AI systems how to behave.

The crawler found 2,615 domains with crawler-policy signals and 382 explicit mentions of AI crawlers such as ChatGPT-User, ClaudeBot, PerplexityBot, GPTBot, or Google-Extended. In other words, robots.txt language is being copied into a model-facing document. Some sites are trying to allow answer assistants while limiting training. Some are trying to name which bots can crawl. Some are mostly cargo-culting syntax from other examples.

Citation control was another major signal. The strict classifier found 337 domains with strong attribution language, while the broader classifier flagged 1,507. That group included domains like 10xtravel.com, 11sight.com, acilearning.com, lawpreptutorial.com, retailbrew.com, stereonet.com, and blossomflowerdelivery.com. The pattern is easy to understand: if a model is going to answer using your work, you want the answer to cite you, quote you correctly, and send the user to the right page.

The most interesting control-plane signal was MCP. I found 1,797 domains mentioning Model Context Protocol. A large subset looked automated: 933 Wix-generated MCP endpoint files exposed tools like business lookup, site search, visitor-token generation, and site API calls. That changes the mental model. A website is no longer just publishing pages for models to read. It can publish a tool boundary and tell an agent what it is allowed to call.

That tool boundary is the sharpest difference between this layer and older discovery files. A sitemap says "these URLs exist." Schema says "this page contains a product or article." A robots file says "crawl this, avoid that." An MCP reference says "there may be a callable interface here." Even if many current references are thin or generated, the direction is clear: the model-readable web is moving from content discovery toward action discovery.

The same pattern appeared in anti-hallucination language. Only 136 domains matched that flag, but the examples were revealing. Some tell assistants not to invent pricing or features. Some route users to local distributors. Some warn that facts are dynamic. This is the beginning of a more honest answer-engine contract: models will summarize, but sites are trying to tell them where not to improvise.

Surprises

The weird parts were more useful than the clean parts

The clean examples show what the format can become. The messy examples show how the web will actually adopt it.

The largest files were not all from the companies I would have guessed. Yes, PixiJS and E2B made sense: developer documentation is an obvious fit for a model-readable full corpus. But the big-file list also included storage racks, nail supply, Pokemon cards, hotel villas, cookware, dealer sites, and small retailers. That is interesting because retail catalogs are exactly where models need structured context but often lack authority. A product page is designed to persuade a person. A model-readable catalog is designed to preserve the facts after summarization.

I also did not expect so many files to look like partial migrations from older web conventions. Some were basically sitemaps. Some were robots files with newer bot names. Some were product feeds wearing a new filename. Some were tiny policies that read like legal notices. That makes the format hard to evaluate if you expect one canonical shape, but it also explains why adoption is happening: people are mapping an unfamiliar AI problem onto tools they already understand.

The files with the most future value were often the ones that compressed judgment, not content. A link dump says, "Here is everything." A good LLM file says, "Here is what matters, here is what changes, here is what you should cite, here is what you should not guess, and here is where the tool boundary begins." That distinction is going to matter more as agent browsers and coding assistants make real decisions from retrieved context.

This is why I want the next pass to score files by intent. Generated adoption, citation policy, full-corpus docs, commerce feeds, MCP/tool links, and explicit safety boundaries are different phenomena. They all happen to live in the same filename today. A serious index should separate them, because a site that says "cite this page and never invent prices" is doing something very different from a site that accidentally exported 400 stale URLs.

Examples

The domains that made the scan feel real

Averages are useful, but the story clicks when you look at specific sites.

Specific domains made the scan feel less abstract. The best examples were not always the biggest brands. Sometimes a small developer tool had the cleanest model-facing docs. Sometimes a retailer had a surprisingly rich catalog dump. Sometimes a regulated business wrote the most careful policy. The mix is the point: the AI instruction layer is forming around real operational needs, not around one clean standards committee.

Evidence

What this can and cannot prove about OpenAI

The tempting claim is that overlap proves a model scraped and used these files. The honest version is narrower and more useful.

This is still worth calling out because it is one of the most interesting questions readers will ask. The practical answer is: "I found lots of sites trying to influence ChatGPT and other assistants, including files that name OpenAI user agents or ChatGPT-specific feeds. I did not prove OpenAI used any one of them." That sentence is less sensational, but it is defensible.

Method

What I would change before the next 1M run

The crawl was useful enough to show the pattern, but the next pass should treat llms.txt as first-class infrastructure.

Classification here was heuristic. It used the domain name, decompressed llms.txt, and the first chunk of llms-full.txt. That was enough to build category counts, profile counts, language guesses, TLD counts, file sizes, and flags for commerce, MCP, citation, crawler policy, AI use policy, agent instructions, full-corpus links, and anti-hallucination language. It is not a legal or safety classifier, and it is not claiming every risk flag is a risky business.

The next version should capture content type, content encoding, final URL, and redirect behavior beside the LLM files. It should keep the homepage favicon and Open Graph image near each row, because those small visual cues make manual review much faster. It should also separate plugin-generated output from hand-authored files, because the incentives are different. A generated sitemap tells you the ecosystem is adopting the convention. A hand-written policy tells you what a company actually wants models to do.

I would also rank files by "instruction density." A 400 KB link dump can be less useful than a 2 KB policy that clearly names canonical pages, dynamic facts, citation requirements, tool endpoints, and things the model should not infer. Byte size is good for finding full corpuses. It is not enough for finding the files that will actually change an answer.

The shopping angle needs a separate pass too. A commerce feed is not the same thing as a purchasable endpoint. Some sites expose product URLs and checkout pages. Some expose feeds but still require a human browser, account state, regional shipping, age checks, or payment confirmation. The next crawler should score discovery, pricing, cartability, payment handoff, and post-payment control separately. "A bot can understand this product" and "a bot can buy this product" are very different claims.

Finally, the safety and policy side deserves more than keyword flags. The current classifier can tell me that legal, medical, financial, gambling, adult, weapons, or substance-related words appear. It cannot tell me whether a site is licensed, whether an answer would be compliant in a user's jurisdiction, or whether a shopping agent should be allowed to act. That is exactly why the model-facing file matters: the site can publish boundaries, but downstream systems still need governance.

The big takeaway is simple: the web is starting to publish instructions for the machines that summarize it. Some of those instructions are SEO. Some are compliance. Some are product feeds. Some are developer docs. Some are actual tool boundaries. That makes llms.txt easy to dismiss if you only look for perfect, hand-authored examples. At 1M-domain scale, the imperfect version is the important version. It shows where the web is already moving.