In my previous article I pondered a couple of issues that make creating a DevOps team a challenging endeavor. Today I want to discuss the following question:
If you have n development teams and one operations team, how can you create m DevOps teams?
In my experience, this is the situation faced by most teams when they think about integrating operations into their agile software development practices. But before discussing this question, let's look at the pros and cons of the situation. Let's start with the pros.
Why do we want DevOps teams?
I read an apt quote by Liker & Morgan in their book on the Toyota Product Development System:
But there was one big problem with [the approach of organizing people on behalf of their function] -- functional specialists tended to bond and become more wedded to their function and profession than to the company and its products and customers. Their measure of success was how well the functional department performed and how big a budget it garnered. They believed their profession could be the savior of the company; if they ran things, the company would be successful beyond belief. As a result, no function did a particularly good job of working in coordination with the other functions.
In my experience, this translates directly to operations and software development in traditional (pre DevOps) companies. Collaboration is at the heart of every agile methodology and if you don't have real collaboration (aka "working together to achieve a goal") because two departments seek different goals (e.g. uptime vs. number of features released) you won't be able to work effectively.
Another big factor is how teams communicate with one another. Many operations and development teams communicate only through tickets. For modern software development this is not enough, because the environment in which the freshly developed software runs is constantly changing on its own(new versions of application containers like Tomcat or databases like MySQL; changes in system configuration; different performance patterns due to updated hardware; etc.) For this reason the input from operations is needed throughout the whole development process; especially during planning where such changes can be addressed proactively.
A third factor is knowledge. Having someone with a different skill-set in the planning session can bring fresh ideas and different perspectives to the table ("Switching to NoSQL is great, but we have to factor in the cost of adapting our backup- and monitoring-systems.")
Last but not least, we'd like to develop a holistic, value-oriented view on our software development efforts. We don't want to focus on tasks, but on the value generated by those tasks. And because services need to be operational in order to generate value (in contrast to software that gets shipped to the customer), this means we have to have someone in the team who cares about the operational aspects.
Why wouldn't we want DevOps teams?
First of all, isn't it better to have specialists working together in a team? They can exchange ideas, talk in their own language and together get better at their craft. When we move to DevOps we loose that, because an operations specialist now works together with software developers and not with other people from his "profession".
Another aspect is the big structural impact such a change could have in a large organization. In some organizations operations and software development are in completely different branches of the org-chart: i.e. the "lowest" common line manager of an operations team and a software development team, is the CEO of the company! If you want to create DevOps teams in such a structure, there are some tough nuts to crack.
Flexibility is also a valid concern. If you have DevOps teams with one team member per team skilled in "operations", what happens when this person is on vacation?
When talking about a way to move from the scenario posted at the beginning of the article to a structure of DevOps teams, we have to address these concerns.
Moving to DevOps
So, how do we go from n development teams and one operations team to m DevOps teams?
Let's say we have 7 development teams and an operations team comprised of 7 specialists. This is mathematically simple: we create 7 DevOps teams by moving each member of the operations team to the 7 software development teams. This will allow us to reap the benefits discussed before. But what about the concerns raised above?
A standard practice in scaled agile environments (and that's what we're dealing with when we have 7 teams), is to have communities of practice (see e.g. Succeeding with Agile by Mike Cohn) ). A Community of Practice (CoP) is a group of people of the same "profession": you can have Java CoPs, Web-UI CoPs, Testing CoPs, etc. Goals for these groups are to define and install "best practices", to solve difficult problems together and to generally foster learning. CoPs are typically self-organizing (just like the agile teams): they decide for themselves how they want to run their respective community.
To give a concrete example of how such CoP could look like, let's consider a Java CoP in our example setting. Let's say the Java community consists of ~25 developers and they all meet every other Friday for two hours. On one of these Fridays they discuss upgrading the JVM to the latest version and agree that they want to test the new version for two full sprints on one of the continuous integrations servers, before rolling it out to production. Next, they talk about aligning on the currently used version of the Apache Commons library, as someone discovered that 5 different versions are currently deployed throughout the project. It is agreed that the all continuous integrations shall be configured in a way that the build breaks if multiple versions of the same library are detected. Tom picks up this task. (Later on, he will put this on the product backlog of his team.) Last but not least, one of the developers gives a short presentation on a new library she learned about at a conference last week and invites all the teams to give it a try in one of their next sprints.
Coming back to our 7 DevOps teams, what we would have in addition to them is (at least) an operations CoP. This group is comprised of all operations specialists from the DevOps teams. They define rules and practices that transcend the individual team. They could, e.g., define a monitoring interface for all teams to use, so that the company can utilize a unified monitoring system. It's noteworthy, that these CoPs not only influence the individual agile teams, but that the teams also influence the CoPs, because every team is represented. This is very different to the traditional approach of a (system) architecture team designing something like a unified monitoring system and then "forcing" the teams to use that. The CoP approach maintains the autonomy of the agile team.
The idea of an operations CoP addresses the first concern discussed earlier: How do operations specialists collaborate if they're distributed across agile teams. It also addresses the concern of the big organizational impact, because a CoP is not an organizational unit. It does not appear in an org-chart.
We now still have the open concern about "flexibility": what happens when one of the operations guys (or gals!) is on vacation? In the original scenario (7 software development teams and one operations team) the remaining 6 people just take on the extra load when one member of the operations team is on holiday (hopefully they had some slack before). Or they push back to some of their customers (the 7 development teams) due to their reduced manpower.
For DevOps the solution is unfortunately not that straight forward, but the concern is still addressable (IMHO). Basically there're two options: 1. a developer from the team takes over the tasks of the operations specialists with guidance from the CoP (where he or she becomes a temporary, honorary member ;-)) or 2. an operations specialist from another DevOps team takes over. The second approach is a bit more messy, as the two DevOps teams in question now have to plan together, because they effectively share a team member. In addition to being easier, the first approach has the benefit of further spreading operational skill and knowledge which increases flexibility in the long run.
Even if we could not completely address all concerns (but 2.5 out of 3 is not too bad), I think that the stated benefits of going the DevOps route and "resolving" an operations team into the (former) software development teams, outweigh by far the impediments that can surface.