27 Feb 2009

Beware MySQL 5.1 my son

garfield's picture
Senior Architect and Consultant
11

The bugs that bite, the flaws that catch!

Earlier today I ran into an annoying bit of evil with MySQL that I suspect is going to bite other people sooner or later. Hopefully I'll be able to save someone some debugging time by pointing out that a year's worth of MySQL 5.1 is broken under at least some Drupal modules. Here's how.

MySQL does a lot of magic automated things, ostensibly to make it easier to write code that doesn't fail. Unfortunately, as with similar attempts in PHP those can cause problems as often as not.

One in particular is NULL handling. MySQL allows, contrary to the SQL specification, a column to be defined as NOT NULL (NULL values are forbidden), and to have a default value. When you try to write a record using INSERT or UPDATE that would place a NULL value in that column MySQL will silently use the default value for you, assuming that's what you meant.

Now that's all well and good and can be quite helpful at times. However, it is contrary to spec and can lead to odd bugs that go unnoticed because the database isn't giving you sufficient red flags. MySQL 5, therefore, added a "Strict mode". Strict mode, among other things, disables that behavior and causes MySQL to whine loudly if you try to insert NULL into a NOT NULL column. Think of it as the MySQL equivalent of PHP's "E_NOTICE" setting. Sounds great, and helps lead to better code.

Well, except that lots of existing Drupal code doesn't work in Strict mode, because it assumes that magic behavior and just inserts NULLs into the database. It's not always laziness, either. Take, for instance, CCK fields. They define a partial schema record, but it's CCK that does the actual database query. But CCK will write NULL values. If a CCK field is defined to have a "NOT NULL default 0" column, that will fail far up in CCK. I know, it happened to me and it took a day and a half to track down. :-)

So why did it fail? Here's where MySQL 5.1 comes in. (I told you that was the issue, didn't I?) Specifically, this bug. In January of 2008, a bug fix was committed to MySQL 5.1 that caused MySQL to throw an error on a NULL value even if it wasn't in Strict mode. The net result is that suddenly Drupal modules that rely on that behavior will fail in MySQL 5.1.

But wait, there is hope! The bug was fixed. Unfortunately, it only fixed on 17 February of this year, so there are no releases yet that have the fix. The net result is that MySQL 5.1.23 through 5.1.32 (not yet released) is incompatible with at least some Drupal modules due to a bug in MySQL.

Fail!

For now, the best solution is to simply stick with MySQL 5.0 for Drupal 6. Fortunately, Drupal 7 now enables MySQL Strict mode automatically. That means modules will have to handle NULL values properly, not just for MySQL but because Drupal itself will enforce it. As a side effect, we work around that bug in MySQL 5.1 in the process.

OK, so Drupal 7 is still a ways away and we have to deal with the here and now. If your module is affected, dig in and fix it! It should be a simple change now that you know how to find it. And if you get bitten by this bug when running a site, well, good luck downgrading to MySQL 5.0.

This has been a public service announcement.

Tags

Comments

It says here that "Field db

It says here that "Field db columns are now forced to have 'not null' => FALSE", however this does not seem to be enforced.

Another reason...

Another reason not to use 5.1 yet :) Good catch.

-N

Postgres

I don't know why the Drupal community is so stuck on MySQL...I'd be recommending Drupal for our company's rebuild if it worked well with Postgres. Core works with this of course, but until module developers do the same, Drupal will be stuck with an unreliable database. You know your software's screwed when the project's founder personally recommends against a "stable" release. So we only use Drupal for small one-off type sites right now...it's a shame because Drupal's a great piece of software.

Many reasons

MySQL is the dominant Drupal database for many reasons. For one, while issues like this do crop up it is still by far the easiest database to get started on. That also has made it by far the most widely available database. Those same reasons apply to PHP itself, of course. In both cases it creates a positive feedback loop.

It's not that Postgres doesn't have quirks. It has different quirks. The most annoying for me is BLOB handling, which makes writing portable queries more difficult.

That said, a small army of people have been putting an enormous amount of work into making the database layer in Drupal 7 much more robust and portable. We now support all three major open source databases in core (and I think we have 100% test coverage on all of them now, or else we're really really close on the Postgres front), and the API Is much richer and allows us to do a lot more database-specific optimization under the hood. Hopefully that means it will be easier for the run of the mill MySQL-centric Drupal developer to write Postgres or SQLite-compatible code just by accident; the vast majority of developers do not even have Postgres running, much less know how to test out their code against it.

We can always use help on that front! There are a number of Postgres-related patches needing work or review, and Postgres experts are in short supply. Please add to the supply!

For those who will be at Drupalcon next week...

One great way to learn more about Drupal 7's new database layer and how you can help make sure that it works well with PostgreSQL and other databases is by checking out Larry's Drupal Databases: The Next Generation session.

Lack of support

As mentioned on http://drupal4hu.com/node/64 there is no proper PostgreSQL support. Period. Tried, failed, moved to MySQL ES, happy. We needed to threaten with the removal of PostgreSQL from Drupal to see Damien and Josh Waihi to stand up as maintainers. Now, MySQL 5.1 is not yet ready it was released but not ready so what? Does not make MySQL 5.0 broken.

Oh and want to hear something that I consider broken in PostgreSQL 8.3? It's continuing on the fine tradition that it tries so hard to be theoretically correct that it's practically unusable. This time, JOINing on different types were deliberately broken. Thanks...

"it's practically

"it's practically unusable"

We manage to use it very well. I agree strongly with a comment from the post you linked: While the majority of Drupal installations are on MySQL, the biggest ones will potentially be on Oracle, Postgres, DB2, and MS-SQL. So while the numbers favor MySQL, it's important to remember that Drupal will never capture any part of the market for high-demand sites unless it offers choice in database, and these are the sort of sites that can take Drupal to an entirely new level by sponsoring contrib projects and improving core. And maybe we'd see more quality admin themes :^)

PostgreSQL emulation

Granted this tip isn't exactly like running PostgreSQL, but it's a little closer and includes the MySQL strict mode. MySQL 5 has a SQL mode called POSTGRESQL that includes STRICT_ALL_TABLES. For people who don't have my.cnf access, it can be set at runtime like SET [GLOBAL|SESSION] sql_mode='modes'

Fixed in the last two releases:

Current version is 5.1.33, so it's been fixed in the last two releases: http://dev.mysql.com/doc/refman/5.1/en/news-5-1-32.html

Patch to devel module for sql_mode switching

I wrote a patch to the devel module to make a sql_mode switcher after reading this.

Bah humbug MySQL!

Thank you SO MUCH for this information! I've been so frustrated trying to track down why my Drupal site wasn't working on my host's server when it worked perfectly on mine -- now I know! And at least for me, the solution is not even that complicated -- they're moving the site to a server that uses 5.0.