Yoast SEO constructs and optimizes various HTTP headers, in order to manage indexing, assist with debugging, and compliment other functionality.
It's sometimes desirable to prevent 'non-page' resources (resources like PDFs or XML files, which don't support HTML
<head> mark-up) from being crawled or indexed by search engines.
When the content management system serves these resources, we can manipulate their HTTP headers to provide crawl and indexing directives via an
x-robots-tag HTTP header.
We do this in the following cases:
- All XML sitemaps return a
noindex, followdirective. NB, this does not affect sitemap consumption or processing.
- XMLRPC files return a
When our redirect manager executes a redirect, we pass a value of 'Yoast SEO Premium' to WordPress'
wp_redirect() function. This adds a
x-redirected-by header to the response, which can be helpful when debugging.