Back in March when we announced Gravatar profiles, your profile page remained private so that only you could access it. Now that you've had a chance to update your information and make sure that only details you want public are there, let's open things up. It's been just over 2 months and more than 300,000 people have updated their details so it's time to do what we've been really excited to do since we first announced profiles: As of right now, … Read More

via Gravatar Blog

Launching products is one of the hardest things companies do. Most companies pour months of work into making sure everything goes right at a launch – the features are right, the marketing is ready, the press is primed, the product is solid, etc. But a new breed of companies are doing things very differently. Instead of optimizing product launches to go as perfectly as possible, they optimize to have them go as quickly as possible.

Let’s compare and contrast:

#1 Optimize for perfection: Features are carefully analyzed and planned, progress is reviewed at multiple stages with various stakeholders, multiple development and testing cycles, launch dates are carefully planned out and coördinated. Result: Product is released every few months.

#2 Optimize for speed: Features are broken into smallest possible pieces, code is incrementally developed, tested and launched, launch dates are fluid, products can be updated in seconds. Result: Product is released continuously. The emerging term for this is continuous deployment.

People will tell you that it’s easy to be fast when your product team is tiny and you have just a few customers, but when you grow you will need to put product development processes in place that will slow things down but help you prevent chaos. One of our goals at Automattic is to prove that particular piece of conventional wisdom wrong. To do so, we have continued to invest in high-speed product releases as our team and customer base have grown. Here is how we do it:

Everyone in our company has access to a deploy button that releases the latest checked in code to about 400 production servers in our web tier in less than 30 seconds (across 3 data centers). Deploys are based on SVN and optimized for fast atomic updates and reverts. Each developer has a sandbox testing environment (running on a XEN virtual machine) and access to detailed debug tools that show memory usage and query and page generation times by blog theme, language, and data center. We develop against live production databases which is possible because each blog has its own set of tables.

This setup allows us to release production code easily and very quickly. On a busy day we do dozens of releases and we’ve structured the other parts of our company to keep up with this pace. For example, we use WordPress (of course) to power our support system which allows anyone in the company to write or adjust a support document on the spot for a new feature. Similarly, we announce updates exclusively on our WordPress.com news blog. The lead person behind a new feature writes the blog post and publishes it alongside the product release. We’ve also developed a staged release system that allows us to release new code to the site but only make it visible to people inside the company. This helps with features that need a little extra testing, documentation, or marketing support.

The pushback I usually get when describing this model is that it’ll never work because things will break at scale. The counter argument is WordPress.com which is very big (250M unique visitors per month), runs blogs for millions of publishers including some of the biggest media brands in the worlds, has an industry leading uptime record, and continues to innovate and evolve at a rapid pace.

Here are 5 reasons why the continuous deployment model works and you should try it for your company:

1. You will be more agile: Pushing code quickly means you can launch features quickly, but maybe even more importantly you can also fix things quickly. By lowering the cost of fixing a mistake from days to seconds, you lower the fear of making mistakes which in turn increases your product development velocity.
2. Product people will love you: What do you think is more appealing to a great product person: A 6 month process of building, tweaking, and justifying a product, or a way to get a new idea in front of customers every day? If you can combine the thrill of instant results with the satisfaction of reaching a large number of customers, you’ve created a very appealing product development environment.
3. Customers like momentum: One of the surprising results of continuous product deployment is that the the old idea of users wanting a perfect product and not being tolerant of flaws is wrong (at least on the web). It turns out that users are forgiving of flaws if they know that they will be fixed quickly. And they tend to like new features better if they show up at a regular pace every few days instead of one giant new product release that changes lots of things around every 6 months.
4. No more gatekeepers: The quest for product perfection leads companies to create lots of gates a product has to pass through before it can be released – user studies, UI signoffs, legal reviews, executive approvals, QA tests, support training, and so on. By making product releases radically smaller and faster, these gates either go away or they can be re-organized to be more agile. For example, writing a support document for a single new feature that gets launched today is a lot easier than planning support for the rollout of dozens or hundreds of changes in a big release.
5. Marketing people will, um, hate you: OK, that’s not really a positive. Or maybe it is. Marketing and PR people (and possibly lawyers) tend to not like the notion of continuous deployment. They are trained to bundle up lots of new features into a big release that is revealed to the world through a carefully orchestrated press and partner roll out that’s designed to break through the noise of everyday distractions. 16 product releases a day simply does not work for that model. The good news is that marketing can adapt. Continuous deployment can lead to continuous chatter by thousands of customers and bloggers which ends up being very valuable, probably more so than that one article in a Big Newspaper every 6 months when you do a major release.

Month long product release cycles made sense when it took a while to package up and distribute software and get everyone to upgrade. On the web, packaging and shipping software can be done in seconds and upgrading is automatic next time someone reloads your site. This makes it possible to radically change the software deployment model from carefully planned and tested, occasional releases, to pushing new software to a web site all day long. I’d like to submit WordPress.com as an example of how this new model can be scaled successfully in the hope that more companies will try it out and embrace it.

I’ve seen other fast growing web startups like SocialCast use continuous deployment with great success, please leave a comment if you know of other examples.

Further reading: Eric Ries has written several in-depth posts about the continuous deployment model.

Here are my top 5 reasons why you should consider the distributed model for your company:

1. Your employees will love it: I can’t overstate how much quality of life people get out of working for a distributed company. You get the best of working remotely – flexible hours, no commute, a personal work environment, much more time with friends and family – without the typical downsides – guilt about being away from the office, or missing out on hallway discussions. In addition, your employees get to live where they want, not where the job market dictates. We’ve had several people join Automattic and then move to places where they always wanted to live. We’ve also had people travel extensively while working from the road. This ability to move without having to worry about getting a different job is very freeing (even if you don’t end up moving, just having the option is nice). I’ve heard from many of our “Automatticians” that they simply can’t imagine going back to a “commute & cubicle” type job.
2. You can hire great people wherever you find them: Once your company is untethered from one physical location, your pool of available job applicants becomes the entire world. You can hire anyone who fits the culture and mission of your company wherever they live. You also get to better insulate yourself from the competition and ups and downs of a particular local job market, and you’ll automatically get better coverage of multiple time zones and languages when your team is more distributed. In our case, the first 4 employees were spread apart pretty widely. First was Donncha in Ireland, then Andy in Texas, Matt in San Francisco, and Ryan in San Jose (I was next, also in San Francisco). Those first 4 guys had already been working together on the WordPress open source project, so it was natural for them to join up and keep working remotely the way they had. And then we just kept adding people in this fashion, often working together first through an open source or consulting project, then joining forces full-time if things work out for both sides. BTW, we use AdminiStaff to deal with the payroll and tax complexities of hiring people all over the place.
3. You will use better communication tools: Communications is a challenge for every company and one that’s amplified for distributed ones because the communication channels are more narrow – a chat conversation is simply not as rich as a real life one. But there are advantages as well. A chat conversation can be archived, searchable, and visible to the entire team, whereas in person conversations in meetings and hallways are often lost to the ether. Being distributed is a good excuse to abolish inefficient meetings, conference calls, and email silos, and get the whole team to use better online collaboration tools. In our case, IRC was our preferred tool for the first year. It’s a real-time chat room for the company that we enhanced by keeping logs (to have searchable archives) and by running bots to automatically publish things like code commit notices into the chat stream. When the team got to about 15 people, the IRC channel got too busy and we split into several channels. We then took this concept a step further and developed a real-time group blogging theme for WordPress called P2 (similar tools are SocialCast and Yammer). P2 provides an activity stream for every project going on in the company. Today, we still have our IRC channel for real-time group chat, but the majority of company communications takes place on a couple dozen P2s (Matt has a great write-up on how P2 changed our company). We also use Skype and email, but only when necessary for one-on-one communication. Anything that isn’t strictly private, we push to P2s to make sure there’s an archive of the information accessible to everyone in the company.
4. You can still be social: Probably the biggest disadvantage to being distributed is the lack of social interaction. Online tools help make up for some of this, but most people like to spend some time together to make their work experience more enjoyable. The good news is that there are ways to compensate for this. We took inspiration from the MySQL team and started having in person meetups for the whole company every 6 months. These meetups last one week and we’ve had them in places like Arizona, Mexico, Canada, Stinson Beach and Colorado. Given how intensely we work together every day, there’s a lot of pent-up excitement by the time we meet in person and the week tends to fly by. During the initial meetups we just spent time hanging out and working our regular jobs side by side for the week. Along the way we refined the model. We now prepare for meetups by thinking up team projects that can be built and launched in a week. This turns our meetups into a kind of “hack week” with 2-3 person teams working on projects and demoing and launching them at the end of the week. We also spend a lot of time socializing, having fun excursions (hiking, golfing, go-kart racing, etc), and doing 10 minute lightning talks followed by Q&A (short talks by anyone in the company on whatever topic they feel like sharing). As we grow, we continue to look for ways to push the envelope on getting people together. We’ve started encouraging teams within the company to have their own mini-meetups, to come work from San Francisco for a while, and to congregate at various WordCamps.
5. Your offices will be more fun: A distributed company doesn’t have one, large centralized office but there are other office options available. Team members often work from home (or really from anywhere), but if there are several people near each other that’s a good reason to start up a co-working space. The key to those spaces is that they’re not permanent offices. No one has a desk that they have to go to every day. People come and go when they want and bring whatever they need to do the work – typically a laptop. It’s a great environment for social interaction (both within the company and with the larger business community in a given area) but it doesn’t undermine the rest of the distributed company because overall company decision-making and communication still happens online, accessible to everyone. We started Automattic with no office and stayed that way for about 18 months. After a while we noticed some weird looks from larger partners because we kept asking to meet at their place or at a coffee shop. So we rented a space on one of the piers in San Francisco for meetings and events. It’s set up like a “lounge” without any permanent desks or offices. Those of us who live in or are visiting San Francisco use it as a co-working space, and we use it often for WordPress meetups and make it available to other startups and tech organizations for events in our area.

So there it is. I know that the distributed model feels very strange to business people who are used to the traditional, centralized way of running a company. But I’m here to tell you that it works. It might even work a lot better than the traditional model for certain types of businesses. After all, distributed systems tend to work well in general (the internet itself being a prime example). If you try this model yourself, I think you will see clear benefits in employee happiness and hiring, and I recommend these core strategies that have worked for us: Have frequent company and team meetups to address the social challenges, use co-working and event spaces instead of traditional offices, and fully embrace new real-time, activity stream inspired communications tools like P2. Good luck!

PS: If the idea of a distributed team excites you, please take a look at our jobs page.

Update: Great follow on post by Bob Patterson about the cost and time saving benefits of a distributed company model.