Skip to content
GeoReady Docs

AI Crawler User-Agent List & robots.txt Rules

AI Bots Reference

Section titled “AI Bots Reference”

Complete reference for AI crawler user-agents and robots.txt configuration strategies.

Citation Bots vs Training Bots

Section titled “Citation Bots vs Training Bots”

This is the most important distinction in GEO robot configuration.

Type What they do GEO impact Example
Citation bots Crawl and index your content for use in AI-generated answers 🔴 Critical — blocking these = not being cited OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Applebot
Training bots Collect data to train AI models (offline process) ⚠️ Optional — blocking doesn’t affect citations GPTBot, ClaudeBot, CCBot

Key insight: You can block training bots (if you have IP concerns) while still allowing citation bots. Blocking GPTBot does not prevent ChatGPT from citing you — OAI-SearchBot is the relevant bot for citations. Likewise ClaudeBot is Anthropic’s training crawler, not the one Claude cites through — Claude-SearchBot (index) and Claude-User (on-demand, when a user asks Claude to visit a URL) are.

Complete Bot Table

Section titled “Complete Bot Table”

🔴 Critical Citation Bots

Section titled “🔴 Critical Citation Bots”

These five bots directly determine whether AI search engines cite your site. Never block them.

Bot Vendor Type Purpose Crawl Frequency
OAI-SearchBot OpenAI Citation ChatGPT Search index — determines citation eligibility Daily
Claude-SearchBot Anthropic Citation Claude search-specific crawler On-demand
PerplexityBot Perplexity Citation Perplexity AI citation index Several times per week
Googlebot Google Citation Same crawler/index that feeds AI Overviews — see note below Very frequent
Applebot Apple Citation Siri, Spotlight Search, Safari Suggestions Periodic

OpenAI (ChatGPT)

Section titled “OpenAI (ChatGPT)”
Bot Type Purpose Crawl Frequency
GPTBot Training ChatGPT model training data Frequent
OAI-SearchBot Citation ChatGPT Search citations Daily
ChatGPT-User On-demand Fetched when a ChatGPT user requests a page As needed

Anthropic (Claude)

Section titled “Anthropic (Claude)”

Anthropic’s current published crawler docs (support.claude.com) name exactly these three bots — anthropic-ai and claude-web are no longer listed and have been removed from this reference.

Bot Type Purpose Crawl Frequency
ClaudeBot Training Claude model training Periodic
Claude-SearchBot Citation Claude search-specific crawler On-demand
Claude-User On-demand Fetched when a Claude user asks it to visit a URL As needed

Perplexity AI

Section titled “Perplexity AI”
Bot Type Purpose Crawl Frequency
PerplexityBot Citation Perplexity index and citation source Several times/week
Perplexity-User On-demand Fetched when a user clicks a Perplexity citation As needed

Google AI (Gemini)

Section titled “Google AI (Gemini)”
Bot Type Purpose Crawl Frequency
Google-Extended Training only Gemini Apps and Vertex AI generative model training/grounding Frequent
Googlebot Search + Citation Traditional Google Search and AI Overviews (same crawler, same index) Very frequent

Note: Google-Extended is a robots.txt token, not a separate user-agent. Per Google’s own documentation, it controls only whether content is used to improve Gemini Apps and Vertex AI generative models — it has no effect on Search inclusion or AI Overviews eligibility, which are governed entirely by standard Googlebot access. To opt out of AI Overviews specifically, Google introduced a dedicated Search Console control (Settings → Search generative AI) starting mid-2026; blocking Google-Extended alone will not achieve that.

Microsoft (Copilot)

Section titled “Microsoft (Copilot)”
Bot Type Purpose Crawl Frequency
Bingbot Search + Citation Bing Search index; Copilot uses this index Frequent

There is no separate CopilotBot — Copilot reads the Bing index, so allowing Bingbot = allowing Copilot.

Apple (Siri)

Section titled “Apple (Siri)”
Bot Type Purpose Crawl Frequency
Applebot Search Siri, Spotlight Search, Safari Suggestions Periodic
Applebot-Extended Training Apple Intelligence training data Periodic

Meta (Facebook AI)

Section titled “Meta (Facebook AI)”
Bot Type Purpose Crawl Frequency
meta-externalagent AI Meta AI (Facebook/Instagram AI features) Periodic
Meta-ExternalFetcher On-demand Meta content fetch on-demand As needed
facebookexternalhit Preview Meta social preview and AI features On-demand

Amazon (Alexa AI)

Section titled “Amazon (Alexa AI)”
Bot Type Purpose Crawl Frequency
Amazonbot Search AI Amazon Alexa and search AI features Periodic

Other AI Bots

Section titled “Other AI Bots”
Bot Vendor Type Purpose
Bytespider ByteDance/TikTok AI + Rec TikTok recommendations and AI features
DuckAssistBot DuckDuckGo Citation DuckAssist AI answers
cohere-ai Cohere Training Cohere language model training
AI2Bot Allen Institute Academic Semantic Scholar, research AI
AI2Bot-Dolma Allen Institute Training Allen Institute Dolma dataset collection
xAI-Bot xAI Citation Grok search citations
PetalBot Huawei Search AI Huawei PetalSearch AI (EU/Asia markets)
YouBot You.com Citation You.com AI search index
CCBot Common Crawl Training Open dataset used by many models

robots.txt — Ready to Copy

Section titled “robots.txt — Ready to Copy”

Full GEO-optimized robots.txt block. Replace https://yoursite.com/sitemap.xml with your actual sitemap URL.

# ═══════════════════════════════════════════════
#   AI SEARCH & CITATION BOTS — Allow All
#   GEO-Optimized robots.txt
#   Updated: 2026-02
# ═══════════════════════════════════════════════


# ——— OpenAI ———
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /


# ——— Anthropic (Claude) ———
User-agent: ClaudeBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /


# ——— Perplexity ———
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /


# ——— Google AI (Gemini training/grounding only — does not affect Search or AI Overviews) ———
User-agent: Google-Extended
Allow: /


# ——— Microsoft (Copilot via Bing) ———
User-agent: Bingbot
Allow: /


# ——— Apple (Siri / Apple Intelligence) ———
User-agent: Applebot
Allow: /
User-agent: Applebot-Extended
Allow: /


# ——— Meta (AI) ———
User-agent: FacebookBot
Allow: /
User-agent: meta-externalagent
Allow: /


# ——— ByteDance/TikTok ———
User-agent: Bytespider
Allow: /


# ——— DuckDuckGo AI ———
User-agent: DuckAssistBot
Allow: /


# ——— Cohere ———
User-agent: cohere-ai
Allow: /


# ——— Academic / Open ———
User-agent: AI2Bot
Allow: /
User-agent: CCBot
Allow: /


# ——— Traditional Search (always keep) ———
User-agent: Googlebot
Allow: /
User-agent: *
Allow: /


Sitemap: https://yoursite.com/sitemap.xml

Strategy: Allow Citations, Block Training

Section titled “Strategy: Allow Citations, Block Training”

If you want to appear in AI answers but prevent your content from being used as training data:

# ─── Training bots — blocked ──────────────────
User-agent: GPTBot
Disallow: /


User-agent: ClaudeBot
Disallow: /


User-agent: Google-Extended
Disallow: /


User-agent: CCBot
Disallow: /


User-agent: Applebot-Extended
Disallow: /


# ─── Citation bots — allowed ──────────────────
User-agent: OAI-SearchBot
Allow: /


User-agent: Claude-SearchBot
Allow: /


User-agent: PerplexityBot
Allow: /


User-agent: Bingbot
Allow: /


User-agent: Googlebot
Allow: /


User-agent: *
Allow: /


Sitemap: https://yoursite.com/sitemap.xml

Note: robots.txt is an honor system. Well-behaved bots respect it; scraper bots may not. For actual training data protection, consider additional legal or technical measures.

Verify Bot Access

Section titled “Verify Bot Access”

Simulate a bot’s HTTP request to check what your server returns:

Terminal window
# Simulate OAI-SearchBot (ChatGPT citations)
curl -A "OAI-SearchBot/1.0 (+https://openai.com/searchbot)" https://yoursite.com/robots.txt


# Simulate Claude-SearchBot (Claude citations)
curl -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +https://www.anthropic.com/claude-searchbot)" https://yoursite.com


# Simulate PerplexityBot
curl -A "PerplexityBot/1.0 (+https://perplexity.ai/bot)" https://yoursite.com


# Simulate GPTBot (training only — does not affect ChatGPT citations)
curl -A "GPTBot/1.2 (+https://openai.com/gptbot)" https://yoursite.com/robots.txt

If you get a 403 or 401 for citation bots, they can’t index your site.

Monitor Bots in Server Logs

Section titled “Monitor Bots in Server Logs”

Check if AI bots are actively crawling your site:

Terminal window
# Search nginx access logs for all major AI bots
grep -E "GPTBot|OAI-SearchBot|ClaudeBot|Claude-SearchBot|PerplexityBot|Googlebot|Applebot" \
  /var/log/nginx/access.log


# Count hits per bot (last 7 days)
grep -oE "GPTBot|OAI-SearchBot|ClaudeBot|PerplexityBot|Google-Extended" \
  /var/log/nginx/access.log | sort | uniq -c | sort -rn


# Apache equivalent
grep -E "GPTBot|OAI-SearchBot|ClaudeBot|PerplexityBot" \
  /var/log/apache2/access.log | awk '{print $1, $7}' | head -50

For managed hosting without log access, check Google Search Console → Settings → Crawl Stats — it shows Googlebot and Google-Extended activity. For other bots, there is no equivalent dashboard; server logs are the only reliable source.

Resources

Section titled “Resources”