Automattic lessons

In praise of continuous deployment: The story

The other day we passed product release number 25,000 for That means we’ve averaged about 16 product releases a day, every day for the last four and a half years! How is this possible and why do we release software in this way?

Launching products is one of the hardest things companies do. Most companies pour months of work into making sure everything goes right at launch – the features are right, the marketing is ready, the press is primed, the product is solid, etc. But a new breed of companies are doing things very differently. Instead of optimizing product launches to go as perfectly as possible, they optimize to have them go as quickly as possible.

Let’s compare and contrast:

#1 Optimize for perfection: Features are carefully analyzed and planned, progress is reviewed at multiple stages with various stakeholders, multiple development and testing cycles, launch dates are carefully planned out and coördinated. Result: Product is released every few months.

#2 Optimize for speed: Features are broken into smallest possible pieces, code is incrementally developed, tested and launched, launch dates are fluid, products can be updated in seconds. Result: Product is released continuously. The emerging term for this is continuous deployment.

People will tell you that it’s easy to be fast when your product team is tiny and you have just a few customers, but when you grow you will need to put product development processes in place that will slow things down but help you prevent chaos. One of our goals at Automattic is to prove that particular piece of conventional wisdom wrong. To do so, we have continued to invest in high-speed product releases as our team and customer base have grown. Here is how we do it:

Everyone in our company has access to a deploy button that releases the latest checked in code to about 400 production servers in our web tier in less than 30 seconds (across 3 data centers). Deploys are based on SVN and optimized for fast atomic updates and reverts. Each developer has a sandbox testing environment (running on a XEN virtual machine) and access to detailed debug tools that show memory usage and query and page generation times by blog theme, language, and data center. We develop against live production databases which is possible because each blog has its own set of tables.

This setup allows us to release production code easily and very quickly. On a busy day we do dozens of releases and we’ve structured the other parts of our company to keep up with this pace. For example, we use WordPress (of course) to power our support system which allows anyone in the company to write or adjust a support document on the spot for a new feature. Similarly, we announce updates exclusively on our news blog. The lead person behind a new feature writes the blog post and publishes it alongside the product release. We’ve also developed a staged release system that allows us to release new code to the site but only make it visible to people inside the company. This helps with features that need a little extra testing, documentation, or marketing support.

The pushback I usually get when describing this model is that it’ll never work because things will break at scale. The counter argument is which is very big (250M unique visitors per month), runs blogs for millions of publishers including some of the biggest media brands in the worlds, has an industry leading uptime record, and continues to innovate and evolve at a rapid pace.

Here are 5 reasons why the continuous deployment model works and you should try it for your company:

  1. You will be more agile: Pushing code quickly means you can launch features quickly, but maybe even more importantly you can also fix things quickly. By lowering the cost of fixing a mistake from days to seconds, you lower the fear of making mistakes which in turn increases your product development velocity.
  2. Product people will love you: What do you think is more appealing to a great product person: A 6 month process of building, tweaking, and justifying a product, or a way to get a new idea in front of customers every day? If you can combine the thrill of instant results with the satisfaction of reaching a large number of customers, you’ve created a very appealing product development environment.
  3. Customers like momentum: One of the surprising results of continuous product deployment is that the the old idea of users wanting a perfect product and not being tolerant of flaws is wrong (at least on the web). It turns out that users are forgiving of flaws if they know that they will be fixed quickly. And they tend to like new features better if they show up at a regular pace every few days instead of one giant new product release that changes lots of things around every 6 months.
  4. No more gatekeepers: The quest for product perfection leads companies to create lots of gates a product has to pass through before it can be released – user studies, UI signoffs, legal reviews, executive approvals, QA tests, support training, and so on. By making product releases radically smaller and faster, these gates either go away or they can be re-organized to be more agile. For example, writing a support document for a single new feature that gets launched today is a lot easier than planning support for the rollout of dozens or hundreds of changes in a big release.
  5. Marketing people will, um, hate you: OK, that’s not really a positive. Or maybe it is. Marketing and PR people (and possibly lawyers) tend to not like the notion of continuous deployment. They are trained to bundle up lots of new features into a big release that is revealed to the world through a carefully orchestrated press and partner roll out that’s designed to break through the noise of everyday distractions. 16 product releases a day simply does not work for that model. The good news is that marketing can adapt. Continuous deployment can lead to continuous chatter by thousands of customers and bloggers which ends up being very valuable, probably more so than that one article in a Big Newspaper every 6 months when you do a major release.

Month long product release cycles made sense when it took a while to package up and distribute software and get everyone to upgrade. On the web, packaging and shipping software can be done in seconds and upgrading is automatic next time someone reloads your site. This makes it possible to radically change the software deployment model from carefully planned and tested, occasional releases, to pushing new software to a web site all day long. I’d like to submit as an example of how this new model can be scaled successfully in the hope that more companies will try it out and embrace it.

I’ve seen other fast growing web startups like SocialCast use continuous deployment with great success, please leave a comment if you know of other examples.

Further reading: Eric Ries has written several in-depth posts about the continuous deployment model.

81 replies on “In praise of continuous deployment: The story”

To me, the greatest thing about this post is not so much the methodology explained herein (which is cool in itself), but the fact that the (supposedly) non-techy CEO of a very-much techy company can explain in details the hows and whys of such methodology, putting another light on the subject – one that other CEOs will listen to, when often they fail to notice the voice of their own techy advisors.

So congrats Toni for this post, hopefully it will ring a bell in other companies 🙂

Thanks for posting this, Toni. The idea isn't as radical as many people have thought.

As someone responsible for the operations of reasonably large sites (Flickr and Etsy), it's in my best interest to deploy small and frequent, as continuous as possible. Once the secret gets out that this approach is actually *good* for availability, the world will be different. 🙂

Completely agree. At SlideShare, instead of big launches, we release code all the time. Apart from all the benefits Toni described, it reduces risk. Since we are putting out small changes at a time, if it breaks we know what it is and can roll back.

"That means we’ve averaged about 16 product releases a day, every day for the last four and a half years!" Incredible. AddToAny is a very small team with an average of *weekly* release cycles for the past 4+ years. Sure it's primarily a widget and thus requires more QA than other platforms & applications, though a part of me is very jealous. 🙂

Cheers, and keep it up. If you do foresee a theoretical tipping point, please do enlighten, but it seems that continuous deployment will be around for as long as your company's DNA is intact.

It makes sense as long as you don't release a big change all of a sudden through your continuous deployment. Everybody likes continuous development, it shows activity behind a product. A comparison could be made between the Gmail and Yahoo mail. Gmail's Lab features show continuous development where as Yahoo appears to be long dead. Hotmail on the other end is still following the old gated mentality. Either case, what I am ultimately suggesting is that you can be a Gmail but you don't want to release Buzz over it.

Hi Toni,

Great read. How do you find continuous deployment different in terms of open source? Perhaps you follow different practices with

@Tor does about 3 official releases per year. There is also a version called Trunk that updates constantly with the latest development changes. Some people run Trunk on their blog to get the latest features and help debug the software, but the vast majority wait for the major upgrades.

Indeed. We practice continuous deployment with many of our WordPress projects and more than a few of them run trunk during the lead up to a new .org release to pull in a feature we needed for the development of the site.

Great write-up. Thanks.

Evidently, this more-more-more aka change-change-change approach suits most people. I expect that I am in a small minority who have a tough time learning the ropes, but battle through, repressing irritation as much as possible, in the expectation that, once they are learned, the clicks that I personally desire to use will become familiar old buddies. Sadly, it is not so. Some young genius "improves" the set-up.

I have great difficulty in editing and inserting new link widgets for my sidebar now. The old system of drag-and-drop placing of new widgets was much better than the new allocating-numbers process. Fortunately I and my sumpnado blog are banal and trite. My essays at school were too. So I do not matter. I realise that. No, seriously. I am not being modest. Oh, you knew that…

> The old system of drag-and-drop placing of new widgets was much better

The new version of widgets still has drag-and-drop. It sounds like you may have javascript turned off in your browser (which would disable drag-and-drop for widgets)? Feel free to contact support to help you out.

I blogged about 4½ years ago about Improvement as opposed to Change, in response to a company's declaration that "[w]e are committed to consistent improvement, not continual drastic change." I've thought for several decades that that sentiment made enormous amounts of sense.

Practicing continuous integration and, more recently, deployment, makes the need for that distinction clearer and more urgent. If you don't know what changed, you can't find and fix what broke when it does. Applying industrial discipline to what heretofore has been a craft process is a hard thing for many people and teams. We evolve or we die.

Great article, Toni. We have been doing continuous deployment at Plex since our inception in 1995. We formalized the process with our own "deploy" button in 2003, and hit the 25k deployment mark less than two years thereafter. We currently average between 50-100 deployments per day to our massive online ERP system.

We, too, encountered the "you'll have to change when you get bigger" argument. But with 7 million lines of code, 2.6 billion hits per month, and 400 Terabytes of data processed daily, we feel we have proven the argument false. It takes a radically different mindset (not to mention developers that can handle the pace of change and related stress), and many software houses just aren't up to the task. But I believe it will become more and more the norm going forward.

Not only does the continuous deployment benefit us for all the reasons you have already given, but it has also given us a distinct competitive advantage over our competitors, many of who are much bigger than us. We are able to deploy changes, fix bugs, and improve the product at a rate that runs circles around our lumbering rivals.

I could go on all day, but you've already painted a great picture and I don't want to ramble. 🙂

BTW, we have a great Marketing VP, and he loves us. You just have to find the right person who can adapt, and actually use the change to our advantage, which we obviously do.

Great article! Sometimes there may be a reason why the lawyers don't want features released (think: potential liabilities) but getting them involved early in the process and getting them to educate (and thus empower) the developers about the issue should linder if not solve the problem.

A good marketer that adds value to the process will also love it. Marketers who retrench into the perceived safety of the gated model should be re-trained or directed toward the revolving doors 🙂

I am a product manager for a SaaS software company always trying to improve our delivery to customers. We have monthly releases and right now that appears to be our current sweet spot. I agree with almost all of what you said but I have a couple of questions:

1. Do you know or expect different customer expectations depending on whether or not they pay for the product? We charge our customers so I am of the opinion that they will be less tolerant of defects. You stated, “By lowering the cost of fixing a mistake from days to seconds, you lower the fear of making mistakes which in turn increases your product development velocity.” That’s great but it also increases your risk of introducing more defects into a live product. Don’t get me wrong, I have no illusion that our product is free of defects but I don’t want to get more lax on introducing them, I want to be more diligent in finding them prior to release.

2. Does the role of product manager change much in a continuous deployment model? We’re Agile now, I’m just wondering if you see the product manager role being less hands on with the development process or even more so. Trying to find the balance between staying on top of new features as they are developed, working through defects and defining how the product should work versus research, tradeshows, and more strategic planning. What are your thoughts?

An interesting thing we’ve found even with monthly releases: our customer facing staff has a hard time staying abreast of the changes and end up scrambling when the customer initiates the drive to use a feature. I think there are numerous ways we can improve this and it probably has less to do with our release cycle and more to do with our ability to train and communicate change.

I'd appreciate anyone's insight.

Great questions.

On #1, we have paying customers and it seems companies are successfully using continuous deployment for both free and paying customers.

> That’s great but it also increases your risk of introducing more defects into a live product

I was concerned about that in the beginning as well, but it has not played out that way. All code has bugs. The key to continuous deployment is to push out small increments of code. By pushing fewer new lines of code, you actually reduce the number of bugs in every release. Also, fast deploys lead to fast feedback and fast reverts which both help in dealing with defects.

On #2, I do think the role of the product lead or manager changes. This model is less driven by a long term roadmap and more by daily and weekly increments. A key part is the ability of the product lead/manager to break a project into small tasks and assign them to the right people in the right order, all while keeping an eye on the larger picture of where a product is headed.

There are business enabler functions and then there are business critical functions. Whilst a defect in a business enabler function can be tolerated, a defect in a business critical function could potentially make you count less money. Additionally, hosting a website (that performs either of the above two functions) inherently poses a string of security and performance vulnerabilities that the system does not have tolerance for. Our current problems are not only in automation of these variables, but also to keep these checks up to date. I know not of an easy way to do it. Get the automation strong in NFR (and R obviously) testing. Easier said than done 😦 How does WordPress do this magic?

> We charge our customers so I am of the opinion that they will be

> less tolerant of defects

Unfortunately it seems that customers are more tolerant of defects nowadays than they used to be in the past. Look at all the recalls, e.g. in the automotive industry. And in the software industry the quality of shrink-wrapped proprietary software has declined massively and is IMHO worse than WordPress. It takes ages for bugs to get fixed in Adobe Photoshop. If they are fixed at all…

What I like about continuous deployment is that it allows for incremental changes at the User's end.

What I hate about Massive Changes is that you suddenly have Massive Incompatibilities where – for example – people send you docx files your computer can't read, or all the Interface changes so much you fumble around looking for stuff. (If it was a car, you'd probably need to spend a day or two driving around a parking lot to get the hang of the new model.)

Continuous deployment feels like the best of Change and Stability.

Thanks for the post Toni. From a technical side, how does the staged release process work? Maybe an engineer could write a post on it sometime? Would love to learn more.

Quite interesting, indeed. What would interest me: How dou you work with features that in their entirety are not programmed within a day or day fragment? Do you work on those with branches, continually merging forward, or is it just that you don't "think" that way anymore?

Good question. With larger features we try and break it down into smaller pieces and release those along the way. Continuous deployment doesn't mean that everything has to be coded and released in a day, just that things get released as soon as they are ready, rather than saving up lots of features into one big release every few months. That said, we are still working on improving our project management skills to better plan and document larger features (without introducing the dreaded "it wasn't in the spec so we can't work on it" hurdle).

Toni, as a software Finance guy (as a CFO living under the requirements of SOX404/SAS70 world of controls) my first reaction to your comments was "this would never fly in the world of the business critical applications arena, where business controls must be non-compromising." However, as I thought about my software development experiences, with the right discipline and possibly certain review/control gates, your approach not only overcomes so many of the obstacles encountered in ERP solutions, but it also creates a mindset shift and culture shift that far exceeds the risks or issues I could put forth. Thanks for sharing the approach — I will circulate this internally to our portfolio companies as a mandate to shorten release cycles and think "smaller and faster is better".

can you share more about the deployment process from SVN? do you tag the release? how to revert in case of error?

Good write up. Few questions I'd be interested in knowing the answers to:

You don't explicitly mention database schema changes here. I'd be interested in how you deal with this given your partitioning strategy?

What software do you actually use to do the code deployment? (And is any of it available in the wilds of open source?)

At, we auto-deploy the latest checked in code as soon as anything is installed, so no safety net at all. Using this method, we're doing 10-30 deploys per day. It's a totally productive way to move quickly.

We still maintain the concept of 'beta' features, though. They're modeled a bit after Google Labs — optional bits of code that can be turned on or off by user account. This lets us roll out new capabilities to limited exposure and avoid disrupting the core user flow.


As a member of a marketing team (I work for the last commenter, Matt Trifiro), I have to say that the benefits of speed and agility far outweigh any marketing kerfuffles. I have been lucky to be trained by people who think normal is the speed of light. So I have developed a whiplash-proof suit of armor and an attitude of "saddle up, sister." It's a great time. I've never had so much fun.

Hi Tony,

Thanks for this very insightful post. As an application developer working for big clients its often quite a hurdle to get development managers (experienced people, but set in their ways nonetheless) on the bandwagon of concepts like continuous deployment. Hopefully your voice and its ripple effect will make a difference in the years to come.

I will definitely add continuous deployment to my business model. Though late to the blog on this subject I am excited by what WordPress is doing and look forward t building a business relationship with Automattic, and you Toni.

I am not yet ready for prime time with my business venture, and a late blooming entrepreneur as well. However, it seemed appropriate to begin building the relationship now.

How early on should I be connecting with you regarding True Ventures funding opportunities?

Thanks for an inspiring blog. I agree with other replies “This is the why to work”.

As a test leader I would like to know more about how you work with tests and continuous deployment. What kind of test do you perform before deploying and on which level (unit, functional and user)? I assume that you have a high degree of test automation?

[…] A New CEO For Automattic: Notes on my transition from CEO to team lead at Automattic. – In Praise Of Continuous Deployment: A new way of making software (from the early days of – Open Source vs Open […]

[…] A New CEO For Automattic: Notes on my transition from CEO to team lead at Automattic. – In Praise Of Continuous Deployment: A new way of making software (from the early days of – Open Source vs Open […]

Leave a Reply to Continuous deployment impressions at #JFall 2010, NL Cancel reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s