Semrus
SEO Toolby Semrush
SemrushBot
Semrush's crawler used for powering its SEO tools: backlink analysis, keyword research, site audit, and competitive intelligence. It is one of the most active SEO crawlers and can consume significant crawl budget.
Respects robots.txt
Yes
Can be blocked
Yes
Crawl-Delay support
No
Type
SEO Tool
Purpose
SEO research, backlink analysis, and site auditing for Semrush platform
SEO Impact
High crawl volume from SEO tool bots like SemrushBot can consume server resources. Blocking it does not affect your search rankings but prevents Semrush from reporting your site data.
User-Agent String
SemrushBot/7~bl
robots.txt Control
Add "User-agent: SemrushBot" with "Disallow: /" in robots.txt.
Block
User-agent: SemrushBot Disallow: /
Allow (default)
User-agent: SemrushBot Allow: /
Official Documentation
Test your robots.txt against SemrushBot
Check which paths are blocked or allowed for each user-agent