Which AI crawlers should you allow in robots.txt?
Allow the crawlers belonging to the assistants you want to be cited by: GPTBot, OAI-SearchBot and ChatGPT-User for ChatGPT; PerplexityBot for Perplexity; ClaudeBot for Claude; Google-Extended plus Googlebot for Gemini and Google AI Overviews. Blocking any of these removes you from that engine's answers entirely, because an engine cannot cite a page it is not permitted to fetch.
Check for a wildcard rule that blocks everything, since a specific allow for one bot does not rescue the others. A group naming a crawler overrides the wildcard group, and the longest matching path rule wins, with Allow breaking ties.
Blocks are frequently accidental — inherited from a staging configuration or set by a previous developer before AI crawlers existed. It is worth checking rather than assuming.
Allowing a crawler is a business decision, not purely technical: it permits use of your content. The trade-off is being citable versus withholding content from model training and retrieval.
Sources
Last reviewed 2026-08-06. We revisit these when the underlying research changes.
Related
See where you actually stand
Run a free check across ChatGPT, Perplexity, Gemini, Claude and Google AI Mode.
Check your AI visibility