What Is an AI Crawler? The Complete List of Bots to Allow in robots.txt
Not all AI bots do the same job. Training crawlers, retrieval bots, and on-demand fetchers — and which you actually need to allow. Check site free.
"AI crawler" gets used as if it's one thing. It isn't. Confusing the four functional categories below is the most common reason site owners misconfigure robots.txt.
1. Training crawlers
Examples: GPTBot (OpenAI), Google-Extended (Google), and CCBot (Common Crawl, used by several vendors).
Collect content to train future language models. Blocking these has no effect on whether the AI cites you in a live answer today — it only affects future training data.
2. Retrieval and indexing crawlers
These build the index that live search features draw from — OAI-SearchBot and PerplexityBot are the two most common examples. Blocking these directly reduces your odds of being cited in real-time answers.
3. On-demand, user-triggered fetchers
Bots like ChatGPT-User fetch a specific page only when a person directly pastes a link or asks about it in the moment — not a background crawl. Blocking these breaks a very direct, high-intent use case: someone actively trying to read your page through an AI assistant.
4. Agentic and tool-use bots
A newer category: AI agents that browse and act on a user's behalf (filling forms, comparing prices, completing tasks). These are still maturing and their identifying user-agents are less standardized than the other three categories.
The full reference
For the exhaustive, exact-syntax list of all 27 known AI crawlers and how to configure each in robots.txt, see our complete AI bots reference guide.
Further reading
Apply this guide
Find the technical blockers before adding more content.
Run the audit when the guide points to crawler access, structured data, llms.txt, or parsing issues that can stop AI systems from using the page.
- Best for crawlability, schema, robots.txt, and llms.txt work.
- Next metric: pass/fail technical signals before content rewriting.
Frequently asked questions
Is GPTBot the same as the crawler that powers ChatGPT search citations?
No. GPTBot collects training data. OAI-SearchBot is the separate crawler behind live ChatGPT search results. Allowing or blocking one has no effect on the other.
What's an 'on-demand' AI bot?
A crawler that fetches a specific page only when a user directly asks an AI assistant about it or pastes a link — as opposed to bots that continuously crawl the web in the background.
Do I need to configure all 27 known AI bots individually?
Not necessarily — most site owners only need to make deliberate decisions about a handful of high-traffic ones (GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, Google-Extended). See our full reference for the complete list and defaults.
Get the monthly State of GEO report
AI search readiness benchmarks, adoption stats, and the actions that move the needle — delivered monthly. No spam.
By submitting, you agree to receive the monthly GeoReady newsletter: benchmark data from the State of GEO dataset, practical GEO guidance, and product updates. You can unsubscribe anytime. See our Privacy Policy.
