Crawling

The process by which search engine bots systematically browse the web to discover and fetch web pages.

Crawling is the first step in how search engines index the web. Googlebot starts from a set of known URLs (seed URLs from sitemaps, previously crawled URLs, and links discovered during past crawls), fetches those pages, renders them (executing JavaScript), extracts links, and adds new URLs to its crawl queue. This process repeats continuously across the entire web.

Googlebots respect robots.txt rules, crawl-delay directives (though Google uses its own crawl rate algorithm), and HTTP status codes. A 200 response leads to parsing; a 301 leads to the redirect target; a 404 or 410 signals the URL should be removed from the crawl queue; a 500 or 503 causes a retry later.

For SEO, ensuring pages are crawlable means: no robots.txt blocks on important pages, correct HTTP responses, working internal links, and fast server responses. Use Google Search Console's URL Inspection tool to see when a page was last crawled and what Googlebot saw.