Claude
AI / LLMby Anthropic
ClaudeBot
Anthropic's crawler for collecting web content used in Claude AI training and web retrieval. Respects robots.txt and is distinct from the Claude AI assistant's live browsing capabilities.
Respects robots.txt
Yes
Can be blocked
Yes
Crawl-Delay support
No
Type
AI / LLM
Purpose
Training data and web retrieval for Claude AI models
SEO Impact
Blocking ClaudeBot prevents your content from being used in Anthropic's training data. May reduce Claude's ability to cite your content in AI-generated answers.
User-Agent String
ClaudeBot/1.0
robots.txt Control
Add "User-agent: ClaudeBot" with "Disallow: /" in robots.txt.
Block
User-agent: ClaudeBot Disallow: /
Allow (default)
User-agent: ClaudeBot Allow: /
Official Documentation
Test your robots.txt against ClaudeBot
Check which paths are blocked or allowed for each user-agent