Are the crawlers of ChatGPT and Perplexity allowed to read your page?
If a robots.txt or a firewall locks out OpenAI's crawler, the content no longer matters. These blocks are usually unintentional and known to nobody in the business.
What the area does
Search and training kept apart
Independent rules apply to each, and the consequences are opposite. Blocking GPTBot prevents crawling for training — that costs no visibility. Blocking OAI-SearchBot excludes your content from ChatGPT's search answers. A check that lumps the two together gives the wrong advice.
Our own robots.txt parser with the “longest match wins” rule
That is exactly how the crawlers evaluate it themselves. A naive check that takes the first matching line reports permissions that do not apply — and blocks that do not exist.
Blocks that do not work are named as such
Some bots do not obey robots.txt. If a block against them stands there, it is ineffective. Reporting it as effective would be false information — the interface instead says the line is there and does nothing.
The finding comes with the lines to fix it
Where a search bot is blocked, the lines to append to your robots.txt stand next to it, copyable, with the path of the file. For training bots nothing is proposed: a block there costs no visibility, and lifting it unasked would interfere with your decision.
How you can tell
- ✓What is checked is what your robots.txt permits for each individual surface — evidenced, not estimated.
- ✓For each bot it states the operator, the purpose (search or training) and whether it obeys robots.txt.
- ✓The check reads only your robots.txt. It costs nothing and can run as often as you like.
- ✓“Never checked” stands there as its own state. It does not mean something is blocked — it means nobody looked.
- ✓Besides the rule, it is evaluated which bots actually visited your pages.
Where it stops
- ×A permission in robots.txt guarantees no citation. It is the precondition, not the result.
- ×A firewall, a bot protection or an access block can prevent retrieval although robots.txt permits it.
- ×The check does not change your robots.txt. It names the lines; inserting them is up to you or your agency.
- ×Bots that ignore the file cannot be locked out this way — that needs the server level.
Frequent questions
Should I block or allow GPTBot?
That is your decision, and it costs no visibility. GPTBot crawls for training models, not for search answers. Anyone who does not want their texts to flow into a model blocks it — and stays visible in ChatGPT search answers as long as OAI-SearchBot is allowed. The two rules are independent.
Why is my page not visible in ChatGPT although robots.txt permits everything?
Because robots.txt is only one of several hurdles. Common further causes: a bot protection or firewall blocks retrieval, the page delivers its text only via JavaScript, or the content does not work as a source because it says nothing only you can say. Access is checked, the citation measured separately from it.
How often should access be checked?
Continuously. A robots.txt changes when somebody updates a plugin, a bot protection is added or a service provider appends a line. That is exactly why the check is free and repeatable — a one-off finding goes stale with the first change.