Generator

robots.txt Generator

Build a valid robots.txt file with crawl rules for any bot, plus one-click presets to block AI training crawlers like GPTBot, ClaudeBot and Google-Extended. Download or copy in seconds.

What gets checked

  • All standard User-agent directives
  • Allow and Disallow path rules
  • AI training bot presets (GPTBot, ClaudeBot, Gemini, CCBot…)
  • Sitemap URL directive
  • Crawl-delay (Bing, Yandex)
  • Instant copy or .txt download
Also trySite CrawlerGoogle Index Checkerrobots.txt Generatorrobots.txt TesterBroken Link CheckerPage Speed TestCore Web VitalsMobile-Friendly TestXML Sitemap GeneratorRender CheckFull SEO Audit
Crawl rules
Rule 1
AI training bots

Select which AI crawlers to block from training on your content.

Extras
robots.txt
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /private/
Place this file at https://yourdomain.com/robots.txt