Twitte
Social Mediaby X (Twitter)
Twitterbot
X's (formerly Twitter) crawler that fetches Twitter Card metadata when URLs are shared in tweets. It reads twitter:card, twitter:title, twitter:image and renders preview cards.
Respects robots.txt
No
Can be blocked
Partial
Crawl-Delay support
No
Type
Social Media
Purpose
Twitter Card metadata scraping for X/Twitter link previews
SEO Impact
Blocking prevents Twitter Card previews from displaying when your URLs are shared on X/Twitter, significantly reducing engagement on shares.
User-Agent String
Twitterbot/1.0
robots.txt Control
Cannot be reliably blocked via robots.txt. Use server-side rate limiting if you need to restrict traffic.
Block
User-agent: Twitterbot Disallow: /
Allow (default)
User-agent: Twitterbot Allow: /
Official Documentation
Test your robots.txt against Twitterbot
Check which paths are blocked or allowed for each user-agent