What is ClaudeBot?
ClaudeBot is Anthropic's crawler for collecting public web content that may be used to train its models. Blocking it excludes your future content from training; Claude's search uses a separate crawler, Claude-SearchBot.
Anthropic runs three agents: ClaudeBot for training, Claude-SearchBot for search indexing, and Claude-User for fetching a page when a user's question needs it. Each can be blocked on its own in robots.txt.
Anthropic advises against blocking its IP addresses as a way to opt out, because the crawler then can't read your robots.txt at all.
Updated
Guides that cover this
AI crawlers list: every bot name for robots.txt
3 min readTo be cited in AI answers, allow the search crawlers: OAI-SearchBot (ChatGPT), Claude-SearchBot (Claude), PerplexityBot (Perplexity) and Googlebot (AI Overviews and AI Mode). GPTBot, ClaudeBot, Google-Extended and CCBot are about model training, and blocking them doesn't remove you from AI search.
Block AI training but stay in AI search answers
2 min readDisallow GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot in robots.txt, and keep OAI-SearchBot, Claude-SearchBot, Claude-User, PerplexityBot and Googlebot allowed. The full file is below.