The other day we passed product release number 25,000 for WordPress.com. That means we’ve averaged about 16 product releases a day, every day for the last four and a half years! How is this possible and why do we release software in this way?
Launching products is one of the hardest things companies do. Most companies pour months of work into making sure everything goes right at launch – the features are right, the marketing is ready, the press is primed, the product is solid, etc. But a new breed of companies are doing things very differently. Instead of optimizing product launches to go as perfectly as possible, they optimize to have them go as quickly as possible.
Let’s compare and contrast:
#1 Optimize for perfection: Features are carefully analyzed and planned, progress is reviewed at multiple stages with various stakeholders, multiple development and testing cycles, launch dates are carefully planned out and coördinated. Result: Product is released every few months.
#2 Optimize for speed: Features are broken into smallest possible pieces, code is incrementally developed, tested and launched, launch dates are fluid, products can be updated in seconds. Result: Product is released continuously. The emerging term for this is continuous deployment.
People will tell you that it’s easy to be fast when your product team is tiny and you have just a few customers, but when you grow you will need to put product development processes in place that will slow things down but help you prevent chaos. One of our goals at Automattic is to prove that particular piece of conventional wisdom wrong. To do so, we have continued to invest in high-speed product releases as our team and customer base have grown. Here is how we do it:
Everyone in our company has access to a deploy button that releases the latest checked in code to about 400 production servers in our web tier in less than 30 seconds (across 3 data centers). Deploys are based on SVN and optimized for fast atomic updates and reverts. Each developer has a sandbox testing environment (running on a XEN virtual machine) and access to detailed debug tools that show memory usage and query and page generation times by blog theme, language, and data center. We develop against live production databases which is possible because each blog has its own set of tables.
This setup allows us to release production code easily and very quickly. On a busy day we do dozens of releases and we’ve structured the other parts of our company to keep up with this pace. For example, we use WordPress (of course) to power our support system which allows anyone in the company to write or adjust a support document on the spot for a new feature. Similarly, we announce updates exclusively on our WordPress.com news blog. The lead person behind a new feature writes the blog post and publishes it alongside the product release. We’ve also developed a staged release system that allows us to release new code to the site but only make it visible to people inside the company. This helps with features that need a little extra testing, documentation, or marketing support.
The pushback I usually get when describing this model is that it’ll never work because things will break at scale. The counter argument is WordPress.com which is very big (250M unique visitors per month), runs blogs for millions of publishers including some of the biggest media brands in the worlds, has an industry leading uptime record, and continues to innovate and evolve at a rapid pace.
Here are 5 reasons why the continuous deployment model works and you should try it for your company:
- You will be more agile: Pushing code quickly means you can launch features quickly, but maybe even more importantly you can also fix things quickly. By lowering the cost of fixing a mistake from days to seconds, you lower the fear of making mistakes which in turn increases your product development velocity.
- Product people will love you: What do you think is more appealing to a great product person: A 6 month process of building, tweaking, and justifying a product, or a way to get a new idea in front of customers every day? If you can combine the thrill of instant results with the satisfaction of reaching a large number of customers, you’ve created a very appealing product development environment.
- Customers like momentum: One of the surprising results of continuous product deployment is that the the old idea of users wanting a perfect product and not being tolerant of flaws is wrong (at least on the web). It turns out that users are forgiving of flaws if they know that they will be fixed quickly. And they tend to like new features better if they show up at a regular pace every few days instead of one giant new product release that changes lots of things around every 6 months.
- No more gatekeepers: The quest for product perfection leads companies to create lots of gates a product has to pass through before it can be released – user studies, UI signoffs, legal reviews, executive approvals, QA tests, support training, and so on. By making product releases radically smaller and faster, these gates either go away or they can be re-organized to be more agile. For example, writing a support document for a single new feature that gets launched today is a lot easier than planning support for the rollout of dozens or hundreds of changes in a big release.
- Marketing people will, um, hate you: OK, that’s not really a positive. Or maybe it is. Marketing and PR people (and possibly lawyers) tend to not like the notion of continuous deployment. They are trained to bundle up lots of new features into a big release that is revealed to the world through a carefully orchestrated press and partner roll out that’s designed to break through the noise of everyday distractions. 16 product releases a day simply does not work for that model. The good news is that marketing can adapt. Continuous deployment can lead to continuous chatter by thousands of customers and bloggers which ends up being very valuable, probably more so than that one article in a Big Newspaper every 6 months when you do a major release.
Month long product release cycles made sense when it took a while to package up and distribute software and get everyone to upgrade. On the web, packaging and shipping software can be done in seconds and upgrading is automatic next time someone reloads your site. This makes it possible to radically change the software deployment model from carefully planned and tested, occasional releases, to pushing new software to a web site all day long. I’d like to submit WordPress.com as an example of how this new model can be scaled successfully in the hope that more companies will try it out and embrace it.
I’ve seen other fast growing web startups like SocialCast use continuous deployment with great success, please leave a comment if you know of other examples.
Further reading: Eric Ries has written several in-depth posts about the continuous deployment model.
Leave a Reply to Hector MinayaCancel reply