Ask HN: Do LLMs Read Your /llms.txt?
In 2024, there was an initiative to standardize the use of an /llms.txt (1) file to provide web resource descriptions that help LLMs use a website. As part of our security analytics (2) development, we're investigating how bots, especially LLMs, interact with web applications.
What we've observed is that this file is never requested by any LLMs. At most, it was requested by bots from BuiltWith and WebPageTest, but not by major LLMs.
Can you check your logs to see if llms.txt is being autonomously read by any popular LLMs?
(1) https://llmstxt.org/
(2) https://github.com/tirrenotechnologies/tirreno
No comments yet.