AI / GEO

AI robots.txt Generator

Control which AI crawlers can access your site. Block LLM training bots while allowing retrieval agents — or find your own balance.

What gets checked

  • 18 AI and LLM crawlers across OpenAI, Anthropic, Google, Meta, and more
  • Preset modes: block training only, allow all, block all, or selective
  • Distinguishes training crawlers from retrieval agents
  • Sitemap URL injection for complete robots.txt output
  • One-click copy and download as robots.txt
Also tryAI Visibility CheckerAI Readiness CheckerAI Citation Testerllms.txt Generatorllms.txt CheckerAI Crawler CheckerAI robots.txt GeneratorChatGPT Visibility
Quick preset
AI bot permissions
BotCompanyPurposeStatus
GPTBot
GPTBot
OpenAITrainingAllow
ChatGPT-User
ChatGPT-User
OpenAIRetrievalAllow
OAI-SearchBot
OAI-SearchBot
OpenAISearchAllow
ClaudeBot
ClaudeBot
AnthropicTrainingAllow
Claude-Web
Claude-Web
AnthropicRetrievalAllow
Google-Extended
Google-Extended
GoogleTrainingAllow
PerplexityBot
PerplexityBot
PerplexityRetrievalAllow
Bytespider
Bytespider
ByteDanceTrainingBlock
CCBot
CCBot
Common CrawlTrainingAllow
PetalBot
PetalBot
HuaweiCrawlingAllow
Diffbot
Diffbot
DiffbotTrainingAllow
ImagesiftBot
ImagesiftBot
ImagesiftBotTrainingBlock
cohere-ai
cohere-ai
CohereTrainingAllow
AI2Bot
AI2Bot
Allen InstituteTrainingAllow
anthropic-ai
anthropic-ai
AnthropicTrainingAllow
Meta-ExternalAgent
meta-externalagent
MetaTrainingAllow
FacebookBot
FacebookBot
MetaTrainingAllow
Applebot-Extended
Applebot-Extended
AppleTrainingAllow
2 blocked · 16 allowed
Sitemap URL (optional)
robots.txt
# robots.txt — generated by SEOCheckPilot
# Standard crawlers (always allowed)
User-agent: *
Allow: /

# AI / LLM crawlers — blocked
User-agent: Bytespider
Disallow: /

User-agent: ImagesiftBot
Disallow: /

# AI / LLM crawlers — allowed
User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-Web
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: CCBot
Allow: /

User-agent: PetalBot
Allow: /

User-agent: Diffbot
Allow: /

User-agent: cohere-ai
Allow: /

User-agent: AI2Bot
Allow: /

User-agent: anthropic-ai
Allow: /

User-agent: meta-externalagent
Allow: /

User-agent: FacebookBot
Allow: /

User-agent: Applebot-Extended
Allow: /
Upload as robots.txt to your site root at example.com/robots.txt. The file must be served with Content-Type text/plain.