How Claude Cites Sources: What We Know So Far

Three different bots, three different jobs

Anthropic runs three separate crawlers, and conflating them is the most common mistake we see. Each has a distinct purpose, and permission for one says nothing about the others.

• ClaudeBot handles training only. It crawls public pages to train future models and does not power citations in Claude's answers.
• Claude-SearchBot is the citation bot: it fetches a page specifically to ground a search-enabled answer and cite it back to the user.
• Claude-User runs on-demand, when a person actively asks Claude to open or read a specific URL right now, not as part of autonomous crawling.

The distinction matters because these are independent robots.txt permissions. Blocking ClaudeBot to opt out of training does not block Claude-SearchBot. A site can refuse training inclusion and still be eligible for citation in Claude's search-grounded answers. Treating them as one decision is the single most common misconfiguration.

What actually gets cited

Claude's web search (available in claude.ai and via the API) surfaces source links the same way ChatGPT search and Perplexity do: a query triggers a search, and the answer cites the pages it drew from. What differs is degree, not mechanism: Claude's search tends to trigger more selectively, on queries where live or specific information genuinely changes the answer, rather than defaulting to a broad multi-source search on every prompt. That's an observed pattern, not a documented algorithm. Anthropic hasn't published the exact triggering logic, and it can change without notice.

Claude Projects, llms.txt, and what's actually confirmed

Projects lets a user attach documents and context directly to a conversation. If someone pastes or uploads an llms.txt file as part of that context, Claude can use it the same way it uses any attached document, but this is a property of Projects accepting arbitrary context, not a confirmed Claude-specific ranking or discovery mechanism. No AI vendor, Anthropic included, has confirmed that publishing llms.txt changes citation likelihood in ordinary web search. Publish it because it costs an hour and makes your priorities explicit, not because it's a known lever for Claude specifically.

MCP and WebMCP: the piece that's actually novel here

This is the one area where Claude's ecosystem is doing something distinct rather than parallel to ChatGPT/Perplexity. Claude supports the Model Context Protocol (MCP) for connecting to external tools and data sources — typically a dedicated server an assistant talks to. WebMCP takes the same idea and applies it directly inside a web page: instead of a separate server, the page itself declares the actions an agent can take on it. Claude in Chrome and similar browser-integrated agents are the natural consumer of that declaration. See our WebMCP guide for the mechanics.

What to check first

• Is Claude-SearchBot allowed in robots.txt? That's what actually affects whether Claude can cite you, not ClaudeBot.
• Does the page answer the query directly in the first few sentences? That's the same answer-first structure every answer engine rewards; there's no Claude-specific trick here.
• Is entity and brand identity clear (schema, consistent naming)? Claude relies on the same signals covered in entity-based authority, not a separate entity graph.
• If the site exposes interactive tools, is WebMCP declared? It's the one genuinely Claude-ecosystem-specific signal on this list.

Further reading

• How Perplexity Chooses Its Sources — the closest comparison point for search-grounded citation behavior.
• ChatGPT Search vs Google Search — how a third answer engine's retrieval differs again.
• What Is WebMCP? The Next Layer of AI Discovery — the declarative-actions layer Claude's MCP ecosystem connects to.
• AI crawlers and robots.txt: the 2026 guide — where ClaudeBot, Claude-SearchBot, and Claude-User fit among all tracked AI bots.
• How Google Gemini and AI Mode Cite Sources