Google
Search Engineby Google
Googlebot
The primary Google crawler responsible for discovering and indexing web pages for Google Search. It visits pages, renders JavaScript, and evaluates content quality for ranking.
Respects robots.txt
Yes
Can be blocked
Yes
Crawl-Delay support
No
Type
Search Engine
Purpose
Google Search indexing and crawling
SEO Impact
Critical — blocking Googlebot prevents your pages from appearing in Google Search. Crawl budget is allocated based on PageRank and site quality.
User-Agent String
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
robots.txt Control
Add "Disallow: /" under "User-agent: Googlebot" in robots.txt. Use X-Robots-Tag: noindex for page-level exclusion.
Block
User-agent: Googlebot Disallow: /
Allow (default)
User-agent: Googlebot Allow: /
Official Documentation
Test your robots.txt against Googlebot
Check which paths are blocked or allowed for each user-agent