Yoast SEO 14.0 was a release that needed quite a few patches so far. We're sorry about that. In building Yoast SEO 14.0 we made a mistake: we bypassed
wpdb. That caused issues and errors for some users when they tried to upgrade, which we're very sorry about. This post explains the mistakes we made, and our reasoning. I hope it explains a bit of how this happens, and, for the developers among you, prevents you from making these same mistakes.
Let's start with what we got right, which paves the way for an exciting future:
- We started using a custom database table for our metadata. As posts tend to have a lot of SEO metadata, this sped up our frontend performance significantly.
- To access that data, we started using an ORM (which stands for Object Relation Mapper). Think of that as a more convenient way of mapping the data you use in your application to the data in the database.
- We didn't remove the data from
wp_postmeta, we just copied it to where we could access it most efficiently, but left it around (and even updated it there) for when other people relied on it being there.
- The vast majority of websites upgraded with no problems, errors or issues.
Note that all the stuff described below was done because a portion of our users was affected. At the same time, the vast majority was not affected, and in fact, saw an immediate speed improvement on their sites.
Picking our ORM
In choosing our ORM we were severely limited as we had to support PHP 5.6. The problem here is that while WordPress has only recently increased its minimum requirements to PHP 5.6, the rest of the PHP world considers that version ancient history. We would have preferred to use Doctrine, but unfortunately, due to this PHP 5.6 requirement, this was unfeasible. Which is why we ended up with Idiorm. It's not perfect, but it was still a lot less work than building our own. We also started using Ruckusing for database migrations. This allows us to easily set up and change our database schema.
Idiorm and Ruckusing both had a dependency though: they are built on top of PDO, a way of connecting to the database that is slightly different from the
mysql interfaces WordPress normally uses in
wpdb. And this is where we made the wrong decision: we decided to open a second connection to the database, using PDO, bypassing
We thought we were ready to release our 14.0 update. We'd tested this extensively, across multiple configurations and systems, and had asked hosts and other plugin developers all over the world to test along with us. We can't always catch every obscure edge-case, but we weren't really expecting a lot of problems. Well. We were wrong. Unfortunately, when you have the number of installs we have, even a problem that affects 0.1% of our users becomes an iceberg.
It turns out loads of people have very non-standard DB connections. Ranging from setups where
DB_CHARSET aren't defined, to larger issues where they are running encrypted database connections, which of course our PDO solution didn't know about.
We encountered lots and lots of small little issues. We could have fixed all that, potentially, but we'd be spending a lot of time doing that and new issues would inevitably keep popping up. So on Tuesday we made a very hard call. We released version 14.0.1 with fixes for all the things we could fix immediately, and then started working on the bigger and more complex challenges, which became version 14.0.2.
The fix, 14.0.2
So we started working on a better solution: still using Idiorm, but passing it all through
wpdb. We've forked Idiorm and we've "semi"-forked Ruckusing: both now connect to the database through
wpdb. This was quite a bit of work, but in hindsight, it wasn't as bad as we thought it would be. Had we known we could do it "this easily", we would have immediately made that step and used this approach - but this is new territory; there are few WordPress plugins which extend the platform in this way, at this scale, with this number of unknowns.
The upside of not connecting through
wpdb was that it had none of the
wpdb overhead. On the other hand, we no longer have to open a connection. In our tests, we see mixed results, but overall, not a whole lot of impact. The added positive side effect is that we can once again inspect our queries through our favorite development plugin, Query Monitor.
Next problem: updates not working as they should
The next problem we ran into was a vexing one: some people were updating to Yoast SEO and they had all the new files, but they were reporting errors with files that no longer existed in the plugin. Note that this does not happen if you update through the WordPress admin. That was baffling. In the end, it turned out that the problem was that some of the WordPress site management tools simply copy files over the old files, instead of removing the directory entirely and replacing it with a directory that contains the new version. This caused errors, but it took a while to diagnose what was going on. That's because this isn't how modern hosting or plugin systems should behave, so it never crossed our mind that they would behave in that way. We got there with the kind help of Ipstenu and Otto in the WordPress forums team, though. Note that the upgrade process for plugins is dealt with by WordPress core, not by our plugin, so this was out of our hands. Regardless, we had to find a workaround.
Because of how Ruckusing, which I mentioned above, works, we included files in a directory to run migrations. If the old files in that directory didn't get deleted, they'd reference classes that we had since removed, and everything breaks. To fix that, we simply renamed that directory, which prevents this from happening for now, and released 14.0.3, this afternoon.
wpdb bites back
And then came the final problem, on Thursday 30th April, which caused us to do yet another update. Turns out that when you
wpdb::prepare a query which has
null in it,
wpdb escapes it into
0, which turns out to be a long standing issue in WordPress core. This is typically something we would have caught if we had not fixed this in a patch release, so had we made our decision to use
wpdb sooner, we wouldn't have caught this problem this late. Unfortunately, now it meant that we were storing the wrong values for
nofollow and other robots meta tags in the wrong way. So... We had to do another patch release, 14.0.4 and re-index everybody's sites to make sure we stored the data right. The re-indexing is possible because we left all the data in
wp_postmeta, a decision I'm now very thankful we made.
Is it worth it?
After reading all this, you might think: is this worth it all? And despite the fact that I and a part of our dev team have been working incredibly hard the last 3 days, the answer is still a resounding: yes! We're laying a foundation for a much better future and we're making sites instantly faster. I hope we've caught all the major issues now, something I of course can't guarantee, and once again, I'm sorry if we caused you trouble!