X-Robots-Tag
In one line
X-Robots-Tag is the HTTP response header version of the meta robots directive — useful when you need to control indexing for non-HTML assets like PDFs, images and other binaries.
Going deeper
X-Robots-Tag delivers directives like noindex and nofollow through the HTTP response header instead of an in-page <meta> tag. The big difference is scope: you can apply indexing rules to non-HTML assets — PDFs, images, video, sitemap files — that meta robots cannot reach.
Common real-world patterns include noindexing all PDFs in a directory, nofollowing a whole subtree, or applying different rules per crawler. Because the header can be set at the server or CDN layer, it is often less effort to maintain than per-page meta tags.
One gotcha: when X-Robots and meta robots conflict, the more restrictive directive wins. To avoid ambiguity, pick one location as the source of truth and let the other follow — chasing 'meta says index, header says noindex' bugs through a CDN config is not a good time.
Related terms
Meta Robots Tag
The meta robots tag sits in the page <head> and tells crawlers whether to index it and follow its links — the per-page lever for indexing policy.
SEOIndexability
Indexability is whether a crawled page is actually eligible to be stored in the search index — a page can be crawlable yet still not indexable.
SEOHTTP Status Codes
HTTP status codes are the standard server responses (200, 301, 404, 500 and so on) that tell crawlers how to treat a request — and in SEO they directly steer indexing decisions.
SEOIndex Bloat
Index bloat is the state where too many low-value or duplicate URLs are sitting in the search index, dragging down the site's average quality and indirectly hurting rankings.
SEOJavaScript SEO
JavaScript SEO is the discipline of making sure JS-rendered pages get crawled and indexed correctly — usually via some combination of SSR, prerendering and hydration.