Automattic lessons

In praise of continuous deployment: The story

The other day we passed product release number 25,000 for That means we’ve averaged about 16 product releases a day, every day for the last four and a half years! How is this possible and why do we release software in this way?

Launching products is one of the hardest things companies do. Most companies pour months of work into making sure everything goes right at launch – the features are right, the marketing is ready, the press is primed, the product is solid, etc. But a new breed of companies are doing things very differently. Instead of optimizing product launches to go as perfectly as possible, they optimize to have them go as quickly as possible.

Let’s compare and contrast:

#1 Optimize for perfection: Features are carefully analyzed and planned, progress is reviewed at multiple stages with various stakeholders, multiple development and testing cycles, launch dates are carefully planned out and coördinated. Result: Product is released every few months.

#2 Optimize for speed: Features are broken into smallest possible pieces, code is incrementally developed, tested and launched, launch dates are fluid, products can be updated in seconds. Result: Product is released continuously. The emerging term for this is continuous deployment.

People will tell you that it’s easy to be fast when your product team is tiny and you have just a few customers, but when you grow you will need to put product development processes in place that will slow things down but help you prevent chaos. One of our goals at Automattic is to prove that particular piece of conventional wisdom wrong. To do so, we have continued to invest in high-speed product releases as our team and customer base have grown. Here is how we do it:

Everyone in our company has access to a deploy button that releases the latest checked in code to about 400 production servers in our web tier in less than 30 seconds (across 3 data centers). Deploys are based on SVN and optimized for fast atomic updates and reverts. Each developer has a sandbox testing environment (running on a XEN virtual machine) and access to detailed debug tools that show memory usage and query and page generation times by blog theme, language, and data center. We develop against live production databases which is possible because each blog has its own set of tables.

Automattic lessons

5 reasons why your company should be distributed

[Update: This post was written a while ago, but the content remains timely and accurate. There are further updates and links at the bottom.]

I’ve noticed a new trend in Silicon Valley. More and more startups are beginning life as distributed companies, and investors and partners are starting to accept it as normal. Our company Automattic is distributed, and I’m ready to sing the praises of running a business in this way. BTW, I think distributed (“evenly spread throughout an area”) is a better description than the more commonly used virtual (“nearly real or simulated to be real”) for a company that has people working from all over the place instead of a centralized office. In Automattic’s case, we currently have over 50 employees spread across 12 US states and 10 countries.

Here are my top 5 reasons why you should consider the distributed model for your company:

  1. Your employees will love it: I can’t overstate how much quality of life people get out of working for a distributed company. You get the best of working remotely – flexible hours, no commute, a personal work environment, much more time with friends and family – without the typical downsides – guilt about being away from the office, or missing out on hallway discussions. In addition, your employees get to live where they want, not where the job market dictates. We’ve had several people join Automattic and then move to places where they always wanted to live. We’ve also had people travel extensively while working from the road. This ability to move without having to worry about getting a different job is very freeing (even if you don’t end up moving, just having the option is nice). I’ve heard from many of our “Automatticians” that they simply can’t imagine going back to a “commute & cubicle” type job.
  2. You can hire great people wherever you find them: Once your company is untethered from one physical location, your pool of available job applicants becomes the entire world. You can hire anyone who fits the culture and mission of your company wherever they live. You also get to better insulate yourself from the competition and ups and downs of a particular local job market, and you’ll automatically get better coverage of multiple time zones and languages when your team is more distributed. In our case, the first 4 employees were spread apart pretty widely. First was Donncha in Ireland, then Andy in Texas, Matt in San Francisco, and Ryan in San Jose (I was next, also in San Francisco). Those first 4 guys had already been working together on the WordPress open source project, so it was natural for them to join up and keep working remotely the way they had. And then we just kept adding people in this fashion, often working together first through an open source or consulting project, then joining forces full-time if things work out for both sides. BTW, we use AdminiStaff to deal with the payroll and tax complexities of hiring people all over the place.
  3. You will use better communication tools: Communications is a challenge for every company and one that’s amplified for distributed ones because the communication channels are more narrow – a chat conversation is simply not as rich as a real life one. But there are advantages as well. A chat conversation can be archived, searchable, and visible to the entire team, whereas in person conversations in meetings and hallways are often lost to the ether. Being distributed is a good excuse to abolish inefficient meetings, conference calls, and email silos, and get the whole team to use better online collaboration tools. In our case, IRC was our preferred tool for the first year. It’s a real-time chat room for the company that we enhanced by keeping logs (to have searchable archives) and by running bots to automatically publish things like code commit notices into the chat stream. When the team got to about 15 people, the IRC channel got too busy and we split into several channels. We then took this concept a step further and developed a real-time group blogging theme for WordPress called P2 (similar tools are SocialCast and Yammer). P2 provides an activity stream for every project going on in the company. Today, we still have our IRC channel for real-time group chat, but the majority of company communications takes place on a couple dozen P2s (Matt has a great write-up on how P2 changed our company). We also use Skype and email, but only when necessary for one-on-one communication. Anything that isn’t strictly private, we push to P2s to make sure there’s an archive of the information accessible to everyone in the company.
  4. You can still be social: Probably the biggest disadvantage to being distributed is the lack of social interaction. Online tools help make up for some of this, but most people like to spend time together to feel more connected and make their work experience more enjoyable. The good news is that there are ways to compensate for this. We took inspiration from the MySQL team and started having in person meetups for the whole company every 6 months. These meetups last one week and we’ve had them in places like Arizona, Mexico, Canada, Stinson Beach and Colorado. Given how intensely we work together every day, there’s a lot of pent-up excitement by the time we meet in person and the week tends to fly by. During the initial meetups we just spent time hanging out and working our regular jobs side by side for the week. Along the way we refined the model. We now prepare for meetups by thinking up team projects that can be built and launched in a week. This turns our meetups into a kind of “hack week” with 2-3 person teams working on projects and demoing and launching them at the end of the week. We also spend a lot of time socializing, having fun excursions (hiking, golfing, go-kart racing, etc), and doing 10 minute lightning talks followed by Q&A (short talks by anyone in the company on whatever topic they feel like sharing). As we grow, we continue to look for ways to push the envelope on getting people together and forming connections. We’ve started encouraging teams within the company to have their own mini-meetups, to come work from San Francisco for a while, and to congregate at various WordCamps.
  5. Your offices will be more fun: A distributed company doesn’t have one, large centralized office but there are other office options available. Team members often work from home (or really from anywhere), but if there are several people near each other that’s a good reason to start up a co-working space. The key to those spaces is that they’re not permanent offices. No one has a desk that they have to go to every day. People come and go when they want and bring whatever they need to do the work – typically a laptop. It’s a great environment for social interaction (both within the company and with the larger business community in a given area) but it doesn’t undermine the rest of the distributed company because overall company decision-making and communication still happens online, accessible to everyone. We started Automattic with no office and stayed that way for about 18 months. After a while we noticed some weird looks from larger partners because we kept asking to meet at their place or at a coffee shop. So we rented a space on one of the piers in San Francisco for meetings and events. It’s set up like a “lounge” without any permanent desks or offices. Those of us who live in or are visiting San Francisco use it as a co-working space, and we use it often for WordPress meetups and make it available to other startups and tech organizations for events in our area.

So there it is. I know that the distributed model feels very strange to business people who are used to the traditional, centralized way of running a company. But I’m here to tell you that it works. It might even work a lot better than the traditional model for certain types of businesses. After all, distributed systems tend to work well in general (the internet itself being a prime example). If you try this model yourself, I think you will see clear benefits in employee happiness and hiring, and I recommend these core strategies that have worked for us: Have frequent company and team meetups to address the social challenges, use co-working and event spaces instead of traditional offices, and fully embrace new real-time, activity stream inspired communications tools like P2. Good luck!

PS: If the idea of a distributed team excites you, please take a look at our jobs page.

Update 6/2013: Three years after writing this post, Automattic is still going strong. We’ve grown from 50 to 180 people. What I wrote above all still applies. We’re still fully distributed, we hire globally, our communications are centered on P2s, we do lots of meetups and events. Our hiring has evolved a little, we now do 3-8 week trial projects with all applicants before hiring someone and every one who joins Automattic starts their job with a three week rotation in customer support (this is working very well for us). We’ve organized the company into 6-8 person teams that are designed to be as self-sufficient as possible (i.e. able to build and launch products on their own). I recently gave a talk with more updated info about “the secrets behind managing a distributed team”:

Update 2/2018: Still going strong! Automattic is 668 full-time employees in 60 countries as of January 19, 2018; in 40 US states (plus the District of Columbia); and in 473 cities around the world. Including English, we speak 79 languages, and 49 percent of Automatticians speak more than one language. We’re still 100% distributed, organized as described above, and our business is booming. The model is working so well, we didn’t even need as big a headquarters as we thought. Tools like WeWork, Slack, and Zoom are making it ever easier to run distributed teams and companies. More links about this topic in this post.

Automattic lessons

Automattic anniversary

Last Saturday marked my fourth anniversary as CEO of Automattic. It’s been an amazing time so far. I think 4 years is the longest I’ve ever been with one company. It’s also the length of an undergraduate college education (!) which gave me the idea to make a list of things I’ve learned during the last few years. The list is getting pretty long and is looking like a good set of suggestions for a series of blog posts about “Lessons learned at Automattic (so far)”. I’m hoping to do a post every week or two, starting with some of the topics I often get asked about:

  • Open source business models: How did we structure Automattic as a commercial entity that both benefits from and supports the open source WordPress project?
  • Distributed companies: We are a team of 50 people spread across 12 US states and 10 countries. How do we make that work?
  • Internet scale services: We run multiple web services that see hundreds of millions of visitors a month. How do you reach an audience the size of Amazon’s and AOL’s with a small startup team?

There’s a lot more. Feel free to suggest what you think would be interesting to hear about (and to nag me if I don’t follow-up!).