llms.txt — the file nobody fetches
Since 2024 many agencies have recommended a file called llms.txt that is supposed to explain the website to AI systems. Ahrefs measured it: for 97 % of the 137,210 domains examined, no crawler ever fetched it.
The answer in one sentence
llms.txt is a proposal without support. No major AI provider has stated that it reads the file — and the measurement shows it is practically never fetched. Creating it costs nothing but time; selling it as a measure sells nothing.
The measurement
Why the idea still sounds plausible
The thought is obvious: robots.txt tells crawlers what they may do — so a llms.txt could tell language models what they should know. That sounds like a clean analogy.
The difference: robots.txt is a standard that site operators and search engines agreed on — meanwhile as RFC 9309. llms.txt ist ein Vorschlag einer einzelnen Person, dem bis heute kein Anbieter gefolgt ist. Ein Format wirkt nicht dadurch, dass es sinnvoll wäre, sondern dadurch, dass die Gegenseite es liest.
What works instead
Getting robots.txt right
This format is read. OpenAI states verbatim that a website without clearance for
OAI-SearchBotdoes not appear in ChatGPT answers. That is the only written assurance from a provider that we know of.Making the text readable without JavaScript
Claude does not execute JavaScript. If your text only appears in the browser, Claude gets an empty shell — regardless of which files sit in the root directory.
Being where citations come from
Only around 5 % of AI citations lead to your own domain. A complete Google Business Profile measurably brings more than any file on your own server.
Our recommendation
Create the file if you like — it does no harm. But do not pay anyone for it, and do not let anyone promise you an effect. In our check we explicitly list llms.txt in our check explicitly as refuted, so that nobody mistakes it for a measure.
As of 1 September 2026 · Ahrefs analysis of 137,210 domains