X-Robots-Tag (HTTP header)
An HTTP response header that carries the same directives as meta robots, but works for non-HTML files like PDFs.
X-Robots-Tag is an HTTP response header that delivers the same indexing directives as the <meta name="robots"> tag. Its primary advantage: it works for non-HTML resources like PDFs, Word documents, images, and CSV files — which cannot have HTML meta tags. Configure it in your web server (Nginx, Apache) or CDN.
All the same values apply: noindex, nofollow, noarchive, nosnippet, max-snippet, noimageindex, and googlebot-specific rules. You can target specific crawlers: X-Robots-Tag: googlebot: noindex or set global rules: X-Robots-Tag: noindex.
Common use case: if you have a large PDF library and do not want the PDFs themselves indexed (just the landing pages), set X-Robots-Tag: noindex in the response headers for all .pdf requests. Nginx example: location ~* \.pdf$ { add_header X-Robots-Tag "noindex"; }
Implementation Example
# Nginx configuration:
location ~* \.(pdf|doc|docx)$ {
add_header X-Robots-Tag "noindex, nofollow";
}
# Apache .htaccess:
<FilesMatch "\.(pdf|doc)$">
Header set X-Robots-Tag "noindex"
</FilesMatch>
# Verified with:
curl -I https://example.com/document.pdf