Question 1

Which AI bots does this tool check?

Accepted Answer

The tool checks 16 AI bots: GPTBot and OAI-SearchBot (OpenAI), ClaudeBot and Claude-Web (Anthropic), Google-Extended and Googlebot (Google), PerplexityBot, CCBot (Common Crawl), Bytespider (TikTok/ByteDance), PetalBot (Huawei), Diffbot, ImagesiftBot (Midjourney), Cohere-ai, AI2Bot (Allen AI), and anthropic-ai.

Question 2

What is the difference between training bots and retrieval bots?

Accepted Answer

Training bots (e.g. GPTBot, CCBot) collect content to train AI language models. Retrieval bots (e.g. OAI-SearchBot, PerplexityBot) fetch real-time content to answer user queries. Blocking training bots prevents your content being used in model training; blocking retrieval bots may stop your site appearing as a cited source in AI-generated answers.

Question 3

Do explicit rules override wildcard rules in robots.txt?

Accepted Answer

Yes. If your robots.txt has a specific User-agent: GPTBot block AND a wildcard User-agent: * rule, the GPTBot-specific rule always takes precedence for that bot, regardless of order. This tool correctly applies that precedence — bots are labelled "Explicitly allowed" or "Explicitly blocked" when a specific rule applies.

Question 4

Should I block AI bots?

Accepted Answer

It depends on your goals. Blocking training bots (GPTBot, CCBot) prevents your content being used to train AI models without compensation. However, blocking retrieval bots (PerplexityBot, OAI-SearchBot) means AI answer engines may not cite your site. Many publishers block training-only bots while allowing retrieval bots to maintain AI search visibility.

AI Crawler Checker

What gets checked