The upcoming 14.0 release of Yoast SEO marks the beginning of a new age for the Yoast SEO plugin. In it, we’ve rewritten our entire metadata engine and we’ve built an abstraction we call indexables. We believe this rewrite will boost WordPress + Yoast SEO as a great platform for SEO for many years to come! Please allow me to walk you through the technological advancements and what they all mean.
Think like a search engine
SEO has always been about thinking like a search engine. Search engines want to retrieve as much information as possible, and use it to provide searchers with the best possible answers to their questions. This means we constantly have to ask ourselves; how does a search engine treat information?
Information on the web is addressable via URLs. Anything that has a URL is something that could be discovered, scraped, indexed and shown in the search results. WordPress has posts, pages, custom post types, categories, tags, custom taxonomies, different types of archives, special pages and maybe even more types of content. Do you think a search engine like Google cares about that? It doesn’t really. It just looks for things with a URL, that it can scrape and index.
A better information architecture for WordPress
From an SEO perspective, any type of page in WordPress is simply an indexable object. This is the basic intuition that has led to indexables. In its core, indexables is just a database table that contains metadata and URLs for all indexables on a site. The abstraction normalizes the information architecture for any type of page in WordPress and makes its metadata directly queryable. On top of that we can now easily and economically relate different indexable objects to each other and to other things, such as links, redirects, attachments, and perhaps even schema markup.
This is a huge deal. Links, for example, are references from one information object with a URL (an indexable) to another information object with a URL (another indexable). We’ve been storing links for posts in WordPress for quite some time now. We can now start doing that for any type of page in WordPress. And by relating them to indexables, we could create a graph of all information on a site, right inside WordPress. That would enable us to provide users with all sorts of useful insights about their site’s SEO.
Direct benefits of indexables
But the direct benefits are great already. With Yoast SEO 14.0, we seem to be hitting massive performance gains, have dramatically reduced the cost of change for our metadata functionality, and can deliver much nicer and more stable API’s for third-party developers who want to integrate with us. Last, but not least, Yoast SEO is now fully ready for headless WordPress. Let me quickly take you through these benefits and how they came about.
WordPress offers standard API’s to store metadata for content resources such as posts and terms. The post meta and term meta API’s implement an entity-attribute-value (EAV) model. To WordPress’ advantage, this is a very flexible and open model which allows any developer to easily add a custom field or piece of metadata to a post or category. However, this model also quickly becomes quite slow to query, especially on big sites and with many custom fields. It is part of the reason all SEO plugins have a tendency to slow down a website. They simply have a ton of metadata to output, all of which needs to be queried separately in an inefficient manner.
By moving to a custom table, we move from EAV to a relational model for fetching SEO metadata, which makes it much easier and more efficient to query metadata for any type of page in WordPress. This advantage becomes especially big when you start relating objects to each other. An indexable is now directly mapped to a term or a post. We can now get all of our data in a single query, which makes any request much faster.
Another big performance gain we get from storing the URL with the indexable. Because of this, breadcrumbs can now be generated in a breeze, where in the past this used to take quite some expensive calculations. In a future release, we also plan to generate our XML sitemaps straight from indexables. This will probably make our XML sitemaps the fastest and most reliable on the web, which is especially good news for large websites, for whom generating XML sitemaps has always been a pain.
Lower cost of change
With Yoast SEO 14.0, we’ve completely rewritten our frontend code and the way it is structured. We're moving from a PHP 5.2 compatible procedural style architecture, to an object-oriented architecture which makes extended use of all features that become available with higher versions of PHP (5.6+). We’re starting to use namespaces, Symfony’s dependency injection container, strict separation between pure PHP services and stateful objects like ORM models and value objects.
These tools and strategies have helped us structure our code in a way that makes it much easier to reason about. The benefits of this are legion:
- The code becomes much easier to debug.
- Changes are easier to make because code concepts are better defined and it’s clearer what part of the code is responsible for what.
- The consequences of code changes are easier to oversee, making unforeseen bugs less likely.
- The code becomes easy to unit test, which helps prevent future regressions even more.
- The time to change or fix something in the code drastically diminishes because of the above. This means we can iterate faster on new features and fix bugs more quickly.
In the last few months we could experience these benefits first hand. We’ve seen that our development team has had a much easier time understanding the functionality. They even came up with clever and critical questions about it because inconsistencies were much easier to spot. This has already resulted in the discovery of some small bugs and inconsistencies which we might’ve otherwise overlooked. We’ve also seen dramatic productivity improvements, with developers often reaching twice the productivity or more when working on the new code.
Better API’s through surfaces
Figuring out how to offer a consistent API to integrators has been a longstanding challenge for us. Within a platform like WordPress, any piece of code that gets executed shares the same runtime with all other active plugins and themes. This means that our “public” interface has technically always been available to integrators. Any class, public method or public function could be invoked by third-party code.
Now the question for us has always been: should you use those? We’ve never been fond of plugins relying on our public API’s outside of our filters and actions. This would mean we’d have to keep those API’s backwards compatible forever, to prevent sites from breaking. That would slow us down and would increase the cost of change again. For WordPress itself, it makes more sense to treat all API’s as public. After all, it is the platform that is running all the code. But how about plugins? Something feels off about treating any plugin like a standard library that you control like any other dependency.
I think we’ve now come up with a very elegant pattern for offering developers a sustainable API contract with Yoast plugins. We call it surfaces and we hope other plugins authors will also start adopting this pattern. A surface is basically an object which we explicitly expose for third-party use and which we promise to keep backwards compatible.
To that end we’ve introduced a global
YoastSEO() function. It has direct access to our DI container and returns a top-level surface, which exposes other surfaces via magic getters (which might in turn expose more surfaces). Through surfaces we can expose pure PHP services, repositories and factories straight from the container or we create more toned down objects which expose only a subset of the behavior. While globally available, each surface is an object in itself which you can simply inject in your code, making it possible for integrators to keep their code decoupled from Yoast SEO. Go check out our new surfaces!
With Yoast SEO 14.0, it will become tremendously easy and efficient to generate metadata output for any frontend request. Because of this, we’ve decided to add a simple metadata endpoint which will make it possible for site builders to use Yoast SEO on headless installs. You can now simply fetch the metadata for a page via the REST API and output it anywhere you like.
How to generate indexables for all your pages?
Yoast SEO will work best if it has an indexable stored for every page on your website. The plugin will automatically generate them over time but that might not cover everything. So on top of that, we also provide tools to generate indexables for the entire site at once. Here's what we provide:
- Whenever content or its metadata gets saved, Yoast SEO will add / update an indexable.
- Whenever a page is visited and it doesn't have an indexable yet, Yoast SEO generates and stores an indexable. This only needs to be done once per page. After that we'll just use the indexable and the page load will be much faster. Any page that gets traffic will eventually have an indexable.
- We've added a tool to the Yoast SEO settings to allow site administrators to add indexables for their entire site at once.
- We've added a WP CLI command to allow site administrators to add indexables for their entire site via the command line.
A look into the future
What we’re seeing today is truly only the beginning. Believe it or not, the real benefits of indexables are in what we can build on top of it. We can start delivering features now that will give a whole new meaning to the term sitewide SEO. While not giving too much away here, the first thing we will build on top of this is a completely revised version of our internal linking algorithm. We’ll soon be able to make our internal linking suggestions sitewide and much more accurate, by building a search engine into WordPress on top of indexables. Sounds a bit crazy right? Yet, with indexables and some help of our computational linguists, we manage to make all this relatively simple! More on that later... 😉 For now, check out the API documentation and stay tuned for the upcoming Yoast SEO release and many more goodies to come!