Web Crawlers & Bots Directory
Complete reference for web crawlers, AI bots, and SEO spiders. Learn each bot's purpose, user-agent string, robots.txt behavior, and whether allowing or blocking it helps your SEO.
Search Engine
GooglebotThe primary Google crawler responsible for discovering and indexing web pages for Google S…Respects robots.txtGoogle→Googlebot-ImageGoogle's dedicated image crawler, used for indexing images in Google Image Search. It resp…Respects robots.txtGoogle→BingbotMicrosoft's primary search crawler for Bing, Microsoft Copilot, and related AI products. B…Respects robots.txtMicrosoft→ApplebotApple's web crawler, used for Siri Suggested Websites, Spotlight Search on Mac/iOS, and Ap…Respects robots.txtApple→DuckDuckBotDuckDuckGo's web crawler for indexing content in DuckDuckGo search results. DuckDuckGo is …Respects robots.txtDuckDuckGo→YandexBotYandex's primary search crawler. Yandex is the dominant search engine in Russia and has si…Respects robots.txtYandex→
AI / LLM
GPTBotOpenAI's crawler for collecting training data for its GPT language models. Content crawled…Respects robots.txtOpenAI→ChatGPT-UserOpenAI's bot used when ChatGPT browses the web in real-time to answer user questions. Unli…Respects robots.txtOpenAI→ClaudeBotAnthropic's crawler for collecting web content used in Claude AI training and web retrieva…Respects robots.txtAnthropic→PerplexityBotPerplexity AI's crawler that indexes content for use in its AI-powered search engine. Cont…Respects robots.txtPerplexity AI→CCBotCommon Crawl's crawler that builds open datasets used to train many major AI models includ…Respects robots.txtCommon Crawl→
SEO Tool
SemrushBotSemrush's crawler used for powering its SEO tools: backlink analysis, keyword research, si…Respects robots.txtSemrush→AhrefsBotAhrefs' crawler used for building its backlink index and powering its SEO research tools. …Respects robots.txtAhrefs→MJ12botMajestic's web crawler used for building its backlink index (Trust Flow, Citation Flow). O…Respects robots.txtMajestic→
Archive
Test how bots see your site
Check your robots.txt rules and see which bots are blocked or allowed
Social Media