The exciting new technology that is indexables

The upcoming 14.0 release of Yoast SEO marks the beginning of a new age for the Yoast SEO plugin. In it, we’ve rewritten our entire metadata engine and we’ve built an abstraction we call indexables. We believe this rewrite will boost WordPress + Yoast SEO as a great platform for SEO for many years to come! Please allow me to walk you through the technological advancements and what they all mean.

Think like a search engine

SEO has always been about thinking like a search engine. Search engines want to retrieve as much information as possible, and use it to provide searchers with the best possible answers to their questions. This means we constantly have to ask ourselves; how does a search engine treat information? 

Information on the web is addressable via URLs. Anything that has a URL is something that could be discovered, scraped, indexed and shown in the search results. WordPress has posts, pages, custom post types, categories, tags, custom taxonomies, different types of archives, special pages and maybe even more types of content. Do you think a search engine like Google cares about that? It doesn’t really. It just looks for things with a URL, that it can scrape and index.

A better information architecture for WordPress

From an SEO perspective, any type of page in WordPress is simply an indexable object. This is the basic intuition that has led to indexables. In its core, indexables is just a database table that contains metadata and URLs for all indexables on a site. The abstraction normalizes the information architecture for any type of page in WordPress and makes its metadata directly queryable. On top of that we can now easily and economically relate different indexable objects to each other and to other things, such as links, redirects, attachments, and perhaps even schema markup.

This is a huge deal. Links, for example, are references from one information object with a URL (an indexable) to another information object with a URL (another indexable). We’ve been storing links for posts in WordPress for quite some time now. We can now start doing that for any type of page in WordPress. And by relating them to indexables, we could create a graph of all information on a site, right inside WordPress. That would enable us to provide users with all sorts of useful insights about their site’s SEO.

Direct benefits of indexables

But the direct benefits are great already. With Yoast SEO 14.0, we seem to be hitting massive performance gains, have dramatically reduced the cost of change for our metadata functionality, and can deliver much nicer and more stable API’s for third-party developers who want to integrate with us. Last, but not least, Yoast SEO is now fully ready for headless WordPress. Let me quickly take you through these benefits and how they came about.

Performance gains

WordPress offers standard API’s to store metadata for content resources such as posts and terms. The post meta and term meta API’s implement an entity-attribute-value (EAV) model. To WordPress’ advantage, this is a very flexible and open model which allows any developer to easily add a custom field or piece of metadata to a post or category. However, this model also quickly becomes quite slow to query, especially on big sites and with many custom fields. It is part of the reason all SEO plugins have a tendency to slow down a website. They simply have a ton of metadata to output, all of which needs to be queried separately in an inefficient manner.

By moving to a custom table, we move from EAV to a relational model for fetching SEO metadata, which makes it much easier and more efficient to query metadata for any type of page in WordPress. This advantage becomes especially big when you start relating objects to each other. An indexable is now directly mapped to a term or a post. We can now get all of our data in a single query, which makes any request much faster.

Another big performance gain we get from storing the URL with the indexable. Because of this, breadcrumbs can now be generated in a breeze, where in the past this used to take quite some expensive calculations. In a future release, we also plan to generate our XML sitemaps straight from indexables. This will probably make our XML sitemaps the fastest and most reliable on the web, which is especially good news for large websites, for whom generating XML sitemaps has always been a pain.

Lower cost of change

With Yoast SEO 14.0, we’ve completely rewritten our frontend code and the way it is structured. We're moving from a PHP 5.2 compatible procedural style architecture, to an object-oriented architecture which makes extended use of all features that become available with higher versions of PHP (5.6+). We’re starting to use namespaces, Symfony’s dependency injection container, strict separation between pure PHP services and stateful objects like ORM models and value objects. 

These tools and strategies have helped us structure our code in a way that makes it much easier to reason about. The benefits of this are legion: 

  • The code becomes much easier to debug.
  • Changes are easier to make because code concepts are better defined and it’s clearer what part of the code is responsible for what.
  • The consequences of code changes are easier to oversee, making unforeseen bugs less likely.
  • The code becomes easy to unit test, which helps prevent future regressions even more.
  • The time to change or fix something in the code drastically diminishes because of the above. This means we can iterate faster on new features and fix bugs more quickly.

In the last few months we could experience these benefits first hand. We’ve seen that our development team has had a much easier time understanding the functionality. They even came up with clever and critical questions about it because inconsistencies were much easier to spot. This has already resulted in the discovery of some small bugs and inconsistencies which we might’ve otherwise overlooked. We’ve also seen dramatic productivity improvements, with developers often reaching twice the productivity or more when working on the new code.

Better API’s through surfaces

Figuring out how to offer a consistent API to integrators has been a longstanding challenge for us. Within a platform like WordPress, any piece of code that gets executed shares the same runtime with all other active plugins and themes. This means that our “public” interface has technically always been available to integrators. Any class, public method or public function could be invoked by third-party code. 

Now the question for us has always been: should you use those? We’ve never been fond of plugins relying on our public API’s outside of our filters and actions. This would mean we’d have to keep those API’s backwards compatible forever, to prevent sites from breaking. That would slow us down and would increase the cost of change again. For WordPress itself, it makes more sense to treat all API’s as public. After all, it is the platform that is running all the code. But how about plugins? Something feels off about treating any plugin like a standard library that you control like any other dependency.

I think we’ve now come up with a very elegant pattern for offering developers a sustainable API contract with Yoast plugins. We call it surfaces and we hope other plugins authors will also start adopting this pattern. A surface is basically an object which we explicitly expose for third-party use and which we promise to keep backwards compatible. 

To that end we’ve introduced a global YoastSEO() function. It has direct access to our DI container and returns a top-level surface, which exposes other surfaces via magic getters (which might in turn expose more surfaces). Through surfaces we can expose pure PHP services, repositories and factories straight from the container or we create more toned down objects which expose only a subset of the behavior. While globally available, each surface is an object in itself which you can simply inject in your code, making it possible for integrators to keep their code decoupled from Yoast SEO. Go check out our new surfaces!

Headless WordPress

With Yoast SEO 14.0, it will become tremendously easy and efficient to generate metadata output for any frontend request. Because of this, we’ve decided to add a simple metadata endpoint which will make it possible for site builders to use Yoast SEO on headless installs. You can now simply fetch the metadata for a page via the REST API and output it anywhere you like.

How to generate indexables for all your pages?

Yoast SEO will work best if it has an indexable stored for every page on your website. The plugin will automatically generate them over time but that might not cover everything. So on top of that, we also provide tools to generate indexables for the entire site at once. Here's what we provide:

  • Whenever content or its metadata gets saved, Yoast SEO will add / update an indexable.
  • Whenever a page is visited and it doesn't have an indexable yet, Yoast SEO generates and stores an indexable. This only needs to be done once per page. After that we'll just use the indexable and the page load will be much faster. Any page that gets traffic will eventually have an indexable.
  • We've added a tool to the Yoast SEO settings to allow site administrators to add indexables for their entire site at once.
  • We've added a WP CLI command to allow site administrators to add indexables for their entire site via the command line.

A look into the future

What we’re seeing today is truly only the beginning. Believe it or not, the real benefits of indexables are in what we can build on top of it. We can start delivering features now that will give a whole new meaning to the term sitewide SEO. While not giving too much away here, the first thing we will build on top of this is a completely revised version of our internal linking algorithm. We’ll soon be able to make our internal linking suggestions sitewide and much more accurate, by building a search engine into WordPress on top of indexables. Sounds a bit crazy right? Yet, with indexables and some help of our computational linguists, we manage to make all this relatively simple! More on that later... 😉 For now, check out the API documentation and stay tuned for the upcoming Yoast SEO release and many more goodies to come!

19 Responses to The exciting new technology that is indexables

  1. Steve
    Steve  • 2 years ago

    As someone with limited skills you have made this impossible for me to use. My site has a ton of post and I have no idea how to run that command line at server level to fix this.

    Thanks for making a mess for me to deal with. It maybe better to do this, but I have 100K posts and hopefully I can get some help somewhere.

    • Joost de Valk
      Joost de Valk  • 2 years ago

      Hi Steve,

      you don't have to do anything and the database will "lazy load": it'll slowly fill as posts are visited. So if you'd rather ignore the notice and not do anything, you can. You'll still get all the benefits, it'll just take a wee bit longer.

  2. Mahmudul
    Mahmudul  • 2 years ago

    Any page that gets traffic will eventually have an indexable. - What about no index, no follow pages/posts?

    • Joost de Valk
      Joost de Valk  • 2 years ago

      If they're ever hit, by anything, they will get an indexable.

  3. Collins Agbonghama
    Collins Agbonghama  • 2 years ago

    I programmatically add meta description to some custom post types like so:

    update_post_meta($post_id, '_yoast_wpseo_metadesc', 'meta description here');

    Will this continue to work?

    • Omar Reiss

      Hi Collins, great question! For the time being, yes, postmeta are even still the original source of truth about a post and are what we use to build an indexable. When postmeta is updated through the normal WordPress API’s for that, we sync the indexable. If and when we want to change this pattern we’ll announce this here and on Twitter and give integrators time to adapt.

      • Collins Agbonghama
        Collins Agbonghama  • 2 years ago

        Good to hear my implementation is upgrade-proof.

        Keep up the good work.

  4. Gal Baras
    Gal Baras  • 2 years ago

    Finger lickin' good! Very well thought out. Can't wait for 14.1 ;P

    Are there any breaking changes, e.g. changes to filter/action hooks?

    • Omar Reiss

      Hi Gal,

      We've deprecated a few filters and actions. In short:
      Deprecated filters: wpseo_twitter_taxonomy_image, wpseo_twitter_metatag_key and wp_seo_get_bc_ancestors
      Deprecated actions: wpseo_opengraph and wpseo_twitter

      We're planning another post about filters and actions that we've deprecated, and new filters that we've introduced.

  5.  • 2 years ago

    Great news! I hope that Yoast 14.0 will work smoothly at once.

  6. Jose A.Molina
    Jose A.Molina  • 2 years ago

    For when 100% compatibility with PageBuilders like DIVI?

    • Omar Reiss

      Hi Jose,
      We're currently working on important technical changes to our editor interfaces so we can reuse them to integrate with page builders better. This will still take some months, but it is very high on our list of priorities.

  7. Clint
    Clint  • 2 years ago

    "We've added a tool to the Yoast SEO settings to allow site administrators to add indexables for their entire site at once."

    Curious as to whether this will work without issue for a site like ours that has over 5,000 posts worth of content plus more than 32,000 images. Many times features like this tend to "time out" because they don't anticipate higher load requirements.

    • Joost de Valk
      Joost de Valk  • 2 years ago

      It’s built with that in mind and we have a WP CLI command that can do this too. Documentation for that coming tomorrow!

  8. Lanier
    Lanier  • 2 years ago

    Great! Me and my team are looking forward to yoast 14.0 ... when is the release date?

    • Joost de Valk
      Joost de Valk  • 2 years ago

      April 28th.

    DALE FILHABER  • 2 years ago

    I read all your posts and information because I always want to improve my SEO. I love your stuff.
    My only concern is in readability when the plugin wants certain transition words - some of which just don't flow. There are lots of other available transition words that don't "count" in your readability score. It would be great to improve your list of transition words.
    Question on the new indexability:
    Will the plugin I have (the premium one) automatically do all the very cool improvements you're talking about?

    • Joost de Valk
      Joost de Valk  • 2 years ago

      Yes. You can run an indexation task to speed up the process of getting all the metadata in our system and I’d suggest running that, but outside of that: yes.

      (And this indexation task can be triggered from a notice that will show automatically after the update)

  10. Rehan
    Rehan  • 2 years ago

    Looking Forward to Trying Yoast 14.0
    Thanks, Omar Reiss 🙂