GitHubDownload ZIP
301 chain301 chain
Permanent redirects stacked one after another.
Found by Redirect checker →
302302
A temporary redirect on a page that moved for good.
Found by Redirect checker →
Technical SEOUpdated

X-Robots-Tag

In short

The X-Robots-Tag is an HTTP response header that gives crawlers indexing or snippet directives for a URL, especially useful where HTML meta tags are unavailable.

The X-Robots-Tag is an HTTP response header that gives crawlers indexing or snippet directives for a URL, especially useful where HTML meta tags are unavailable. X-Robots-Tag controls eligible crawler behaviour at the HTTP-response layer; it does not replace a clear URL, canonical and access policy, and robots.txt is a different control.

An HTTP response can carry indexing and link-following directives.

Definition and boundary

X-Robots-Tag controls eligible crawler behaviour at the HTTP-response layer; it does not replace a clear URL, canonical and access policy, and robots.txt is a different control.

Practical workflow

  1. Decide whether the URL should be indexed, shown with a snippet, followed or kept out of a specific crawler’s result set.
  2. Return the directive on the intended final response, not only on an intermediate redirect.
  3. Inspect headers with a request that follows the public route and record the final status.
  4. Check that HTML meta robots and header directives do not conflict.
  5. Retest after CDN, cache or application changes because headers can be rewritten.

FAQ

Yes. The HTTP header can apply robots directives to HTML as well as files such as PDFs.
A crawl block or inaccessible resource can prevent retrieval. Check the actual response the crawler is allowed to fetch.