Is AI blocked from your site?
Check whether GPTBot, PerplexityBot, ClaudeBot, Google-Extended and ten other AI crawlers can reach your pages. This is the one technical check that is a fact rather than a theory: an engine cannot cite a page it is forbidden to fetch.
Common questions
Why does AI crawler access matter?
An answer engine cannot cite a page it is not permitted to fetch. If robots.txt blocks GPTBot, PerplexityBot, ClaudeBot or Google-Extended, you are removed from that assistant's answers entirely — regardless of how good your content is. Unlike most technical AEO advice, this is a fact rather than a correlation.
Which AI crawlers should I allow?
Allow the crawlers belonging to the assistants you want to be cited by: GPTBot, OAI-SearchBot and ChatGPT-User for ChatGPT; PerplexityBot for Perplexity; ClaudeBot for Claude; Google-Extended plus Googlebot for Gemini and Google AI Overviews. Allowing a crawler permits use of your content, so it is a business decision as well as a technical one.
I never blocked anything. Why is a crawler showing as blocked?
Blocks are frequently accidental. Common causes are a wildcard Disallow rule inherited from a staging configuration, a security plugin adding bot rules, or a robots.txt written by a previous developer before AI crawlers existed. A rule naming a specific crawler overrides the wildcard group, and the longest matching path rule wins, with Allow breaking ties.
Does allowing crawlers guarantee I get cited?
No. Crawler access is the floor, not the goal — it makes you eligible to be cited rather than likely to be. The strongest measured correlate of AI visibility is third-party brand mentions, which Ahrefs found correlating at 0.664 across 75,000 brands, roughly three times the correlation of backlinks.
More on what actually moves AI visibility in our sourced answer on the evidence.