A surprising number of sites that ask "why does ChatGPT never mention us" have a robots.txt that forbids ChatGPT from reading them. Usually it was a well-meant line added in 2023 to keep the site out of training data. The crawlers have since split into training, search and browsing agents, and one Disallow written for the first blocks the other two.
Which AI crawlers exist, and what does each one do?
Answer. Each vendor runs separate user agents for training, for its search index and for fetching a page on a user's behalf. Only the last two affect whether you appear in an answer.
The one that surprises people most is Google-Extended. It is a control for whether your content is used in Gemini's training and grounding; Google's documentation states that AI Overviews use the ordinary Googlebot index, so Disallow: Google-Extended neither removes you from Overviews nor protects you from them.
What does this check detect?
Answer. A robots.txt that disallows any of the search or on-request agents (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User), or a blanket User-agent: * block that catches them.
Moonleap fetches /robots.txt, parses the groups, and resolves the rules for each named agent the way the agent would. The finding lists exactly which agents are blocked and by which line. Blocking the training crawlers (GPTBot, ClaudeBot, Google-Extended) is a choice, not a finding; the check leaves those alone.
What is the fix?
Answer. Name the training crawlers you want to block, and say nothing about the search and on-request ones so the default (allow) applies.
A robots.txt that keeps text out of training while staying citable:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: *
Allow: /
Sitemap: https://<your-domain>/sitemap.xml
If you would rather be used for training too (many small sites decide the reach is worth it), delete the first three groups. If your file has a User-agent: * group with Disallow rules for private paths, that is fine; the agents above follow those rules like any crawler. The failure is a Disallow: / under a wildcard, or a specific block on a search agent.
One more thing worth checking while you are in the file: a robots.txt that lists /admin, /dashboard, /internal is a map for anyone curious. Protect those routes with authentication and leave them out of the file; a route that returns a redirect to login cannot be indexed anyway.
How do you verify it?
Answer. Fetch the file and evaluate it for each agent name; then, for ChatGPT specifically, ask it to open your page.
curl -s https://<your-domain>/robots.txt
Read the groups top to bottom: a crawler uses the most specific group that names it, otherwise the * group. For a live test, ask ChatGPT (with search enabled) to summarise a specific URL on your site. If it answers from the page, ChatGPT-User can reach you; if it says it cannot access the page, it is blocked or the page requires JavaScript to render its text.
FAQ
Does allowing these crawlers cost me anything?
Bandwidth, and very little of it. The on-request agents fetch one page when a user asks; the search bots crawl at rates comparable to Bing.
Can I allow search but block training with a single line?
No. The vendors deliberately use separate user agents so that you can. Two groups, one for each, is the minimum.
Will blocking GPTBot hurt my ranking in ChatGPT search?
Not directly. OpenAI's documentation separates the two; blocking GPTBot keeps you out of training data while OAI-SearchBot still indexes you. In practice the sites we see missing from ChatGPT answers blocked all OpenAI agents at once.
Sources
- OpenAI, "Overview of OpenAI crawlers" https://platform.openai.com/docs/bots
- Anthropic, "Does Anthropic crawl data from the web, and how can site owners block the crawler?" https://support.anthropic.com/en/articles/8896518
- Perplexity, "Perplexity crawlers" https://docs.perplexity.ai/guides/bots
- Google, "Google's crawlers and fetchers", entry for Google-Extended https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers
- Google, "AI features and your website" https://developers.google.com/search/docs/appearance/ai-features