We're not an underground success anymore

Should we go mainstream?

Recently, I've had more than one instance where someone either asks whether Drupal is a hobby project, or seems to treat it as such. To be sure, Drupal had its origins as a hobby side project, and those origins are still apparent in its code base, development workflow, architecture, and in the tools used to build and support it. However, that is not what Drupal is now.

Drupal is a leading Open Source Content Management Framework. It powers several dozen of the top 1,000 sites on the Web. Fortune 500 companies dig Drupal. There are easily 1,000 Drupal-centric companies around the world, and thousands more independent developers. Tens of millions of dollars of venture capital have been poured into high-profile Drupal-based companies. Stodgy old analyst firms consider Drupal a visionary alongside Google. Father-of-the-Web Tim Berners-Lee now runs the UK government's open-data initiative on Drupal. Drupal powers the Prime Minister of Jamaica, the King of Belgium, and the White House itself!

In terms of number of contributors per lines of code in the core system, I am fairly sure that Drupal 6 has more contributors per line that the Linux kernel does.

Drupalcamp Chicago 2009 had over 300 attendees, making it better attended than Drupalcon Sunnyvale (my first DrupalCon). In planning for Drupalcamp Chicago 2010, I don't think 500+ attendees is unreasonable. And we're one of the newer, smaller Drupalcamp communities. New York, LA, and the Bay Area have been around longer and are even larger.

Drupal went mainstream a long time ago. I would argue, in fact, that the turning point was the founding of the Drupal Association; the original impetus for which was that enough third parties wanted to support "Drupal" in non-code ways (donating money, donating servers, etc.) but there was no "Drupal" other than Dries Buytaert himself, which didn't work so well, especially when he was a Ph.D student. And that was in 2007. It's now 2010.

To be sure, Drupal is a hobby for a lot of people. Many of those Drupal hobbyists go on to become major core contributors (including yours truly). Many stay as casual contributors or fanboys, and that's fine. But that's different from Drupal itself being a hobby project. It's not. It's a professional, enterprise-grade system with a large commercial ecosystem backing it, and hundreds or thousands of people who depend on it for their livelihood.

Drupal is growing up into a major professional project. Some of that process is painful, and it changes the dynamic of what Drupal is and how we work with it. But for better or worse, it is inevitable.

To be blunt, those asking if Drupal is a hobby project are at least three years behind the times. In Internet time, that's an eternity.

What does it mean to be mainstream?

Dries has already observed some of what it means to become a mainstream success: Acknowledging that we are not our own primary user base is a big one. But it also means acknowledging what we're not good at, and deciding if we're really going to invest the time (and money) to get good at it or if we're going to seek outside help. It means accepting that some of the jokes you make in an underground movie theater just don't play in Peoria, and some of the architectural decisions you make when your main audience is weekend-PHP-hackers running a blog are different than the ones you make when your main audience is professional site developers and institutional content editors.

What have we done?

In some ways we're already adapting to being a professional project.

• The Drupal Association exists in the first place.
• When it came time to redesign Drupal.org, we hired professionals rather than relying on limited in-house expertise or casual volunteer work.
• The Drupal Association has recently moved to partnering with professional events management firms to run DrupalCon, rather than expecting an infinite supply of unskilled volunteers to manage ever larger events at their employer's considerable expense.
• Dries is talking about revising our process some for Drupal 8, especially the concept of having multiple core maintainers who focus on different areas.

But there's still far more to think about.

Oh yeah, like what?

As we continue to adapt to being a professional-grade project, not a hobby project, there's a lot of other hard questions we, as the Drupal community, should be asking ourselves as well. For example...

Why do we still use CVS for version control? Yeah, yeah, I know why. It's a cop-out. CVS is holding us back. Full stop. A lot of very interesting and important development is now happening on GitHub or similar third-party sites, and then eventually (we hope) coming back onto Drupal.org in huge untrackable chunks. Whether it's git, bzr, or whatever, we need to do something. Sitting around waiting for "the community to provide" is not working.

In fact, why are we still hosting our own repository in the first place? Projects with far more resources that we do are migrating to hosted code management such as Gitorious so that they don't have to deal with it. We should be open to that as well, so that we aren't wasting time having volunteers fight with a decades-old system. (We should be paying a great deal of attention to KDE, actually, as they're a very similar project, socially.)

For that matter, why are we still spending time on project.module at all? There are other existing tools that offer much more flexibility, and some are even open source.

If we want to continue to spend resources on maintaining our own home-grown infrastructure, we need to commit to spending those resources. That means time and money, and in some cases a lot of it. Right now, we're not doing that.

The Drupal community hired professionals to design the new Drupal.org, but not to implement it. For that, we relied on "community volunteers". While much progress has been made, we're still running the same tired old design over a year later, and still trying to find unpaid volunteers. We're about to release Drupal 7, which will have a new and modern-looking user interface, but there's still no guarantee that Drupal.org won't still be running a design and information architecture that was built for Drupal 4.5 when that happens.

We need to acknowledge that not all man-hours are created equal. Not everything in Drupal can be, or even should be, approachable and understandable by every weekend-PHP-jockey. That's a laudable goal, to be sure, but complex problems sometimes have complex solutions, and we need to accept that. (As complex as necessary, but no more.)

Software architecture is such a complex problem.

With Drupal being used in more and more mission-critical systems, perhaps it's time to reconsider our long-standing attitude of "we don't babysit broken code" (also known as "if you have a bug, sucks to be you"). It's a great excuse to keep core code lean and mean, but it also means many simple, easy to fix bugs are virtually impossible to find. A robust system needs robust error handling, and "you mistyped something? sucks to be you, hope it doesn't hose your database" is not robust. And yes, that means possibly spending CPU cycles on it.

Most of the highest-profile developers are at least in part sponsored by their employer, but also work a lot of evenings and weekends on their own time. However, Drupal is getting substantially larger, and we cannot rely on volunteer passion and generous employers forever (especially in this economy, and especially when developers have kids, or even grandkids). Will we reach a point where we will have to start paying developers, or at least some of them, just to make sure we have enough skilled hands in the right place? Who can do that, and how would we decide who to pay? I don't know, but it's a question we should be asking.

With Drupal getting larger, do we need more structure to our development process?

Dare I say it, do we need roadmaps?

Maintaining Drupal or a Drupal subsystem is very different than writing code for it. It's closer to management or project management than software development. Should we be looking for more managers and project managers as maintainers? Do we need professional cat-herders, not cats trying to herd other cats?

Should we be providing training in such project management for our core maintainers? Should we even be figuring out what the heck subsystem maintainers do?

Software development and software engineering and architecture are decidedly different problem spaces. Should we have people, formally or informally, who are primarily architects, not code slingers? That's going to mean trusting them and their architectural decisions, something that the Drupal community (and open source folk in general) has not always been particularly good at.

Conclusion

To be sure, not all of the above ideas are good ideas. Some wouldn't work, some are just really really stupid when you think about it, some are infeasible for cultural reasons. But we still need to take the time to ask these questions, because whether we like it or not Drupal is not a hobby project anymore and it cannot go back to being one. We're not an underground success anymore, and we're playing in large national theater chains, not just independent art house theaters.

We cannot stop these changes, whether we want to or not. We can either change with them and direct them, and ourselves, into a direction we want, or we can get run over by them.

The growing up conversation started in Paris. Let's continue it in San Francisco.