Welcome back to our mini-series on managing releases “the DevOps way”. Before we dive into the meat of today’s topic, allow me to explain why I’m ascribing the DevOps moniker to these ideas.
DevOps is an approach to building high-quality, rapidly-delivered software in a way that isn’t painful. DevOps is not a tool. DevOps is not a team. It’s a philosophy, and is best summed up by the CAMS acronym: Culture, Automation, Measurement and Sharing. Last week’s post about cross-team collaboration at release time was all about culture–with a little automation thrown in for seasoning. Fostering a culture of open communication and shared responsibility is a very DevOps-y thing. A tool like Bamboo can level-up your cross-team kung fu, but let’s be clear: the DevOps culture comes from within. It’s you and your colleagues and your collective commitment to making everyone’s work flow smoothly that makes the biggest difference.
Now if someone could just help me down from this soapbox, I’ll dust off my political science degree (as promised) and get on with this blog…
So! Today we’re reading again from the “2 parts culture, 1 part automation” cookbook and talking about getting release candidate builds promoted through our internal environments and out to the customer. Let’s set the scene with the way far too many unfortunate souls are doing this right now.
Let’s say a bug fix has been committed just in time to make the release and has passed the automated tests. Now you need to kick the tires with a bit of exploratory testing on a QA environment. If you’re in a highly-siloed organization, your next move is probably requesting a deploy from the ops team. If you’re lucky, you get to swing by someone’s desk and request it in person or hit the ops team up over IM. If you’re un-lucky, you’ll file a request ticket then obsessively check your email for some sign that your request didn’t fall into a black hole. Either way, your deploy is put into someone’s work queue and will be serviced as soon as they can get to it.
This does NOT mean your sys-admins don’t want to be helpful. They do. Promise. But they’re under pressure to keep environments stable and keep servers secure, and someone in management has decided that the best way to ensure that is for all deploys to be channeled through ops.
“Draconian!” you protest. And you’re right. This is the software development equivalent of feudalism. It’s a long-standing system that serves only a few, at the expense of the many: devs and testers whose work is blocked, as well as the sys-admins who would rather be doing more interesting work than firing off a script and resolving a ticket. This is ultimately unsustainable. Good workers will get fed up and leave, or the organization will fail to keep up with it’s competitors, or both. Enter the revolution.
The obvious response is to give developers and testers full control over deployments, letting operations focus on building out environments and monitoring them. But you know it’s only a matter of time before a build is deployed to a sensitive environment like sales’ demo environment or (God forbid) production with untested changes that result in an emergency. “It was only a one-line change!” says the anguished responsible party. Nonetheless, members of the dev, QA, support and ops teams are now embroiled in troubleshooting instead of doing forward-looking work.
This does NOT mean your developers and testers have no sense of ownership. They do. Promise. But they’re under pressure to get new features out to the customer as quickly as possible, and sometimes that gets (wrongly) interpreted as a license to cut through red tape with a claymore.
“Chaos!” you cry. And you’re right. This laissez faire arrangement has the same problem that Karl Marx found with capitalism as it existed in the mid-1800s: the extreme lack of regulation ends up creating a level of conflict that is every bit as unsustainable as it’s polar opposite. Bring on the evolution.
Yes Virginia, there is a middle ground. It’s probably pretty obvious by now: let developers and testers deploy to dev and QA (and probably load test) environments all they want, but keep production deploys restricted to members of the ops team or (better yet) a cross-team selection of project leaders. In other words, deploys to an environment are in the hands of the people whose work primarily takes place in that environment.
This has two benefits. First, it means you’re allowed to make as many messes as you’re willing to live with and clean up yourself. The people who use the QA environment will normalize on their own threshold for brokenness, while ops, whose threshold for brokenness in production is near zero, can hold the line and push changes only when testing has proven them worthy. Just as important, though, is that bottlenecks are smashed to bits. Self-service deploys mean devs and QA get new builds up on demand, and sys-admins’ queues instantly have a higher interesting-work-to-volume ratio. Just imagine the feudal scenario where every deploy needs a request ticket. Now multiply that agony by 10 because you’re working with people 7 timezones away. But with per-environment deploy permissions, even physical separation matters hardly at all.
“Sensible!” you exclaim. And you’re right. Say what you will about communism*–the principle of “to each according to their need; from each according to their ability” has some merit. Nice one, Uncle Karl. And let’s not ignore the strong element of individual responsibility and participation inherent in this approach. Hey, we like to have it all.
The other thing that should be pretty obvious by now is that our quest for a socio-democratic deployment utopia inspired us to bake per-environment permissions into Bamboo 5. For each application you deploy with Bamboo, permissions for viewing, configuring and executing deploys can be assigned to individual users or groups. It’s a small feature with a big impact (and, apparently, an epic history).
Ready to start democratizing your deployments? Grab the Bamboo 5 beta. UPDATE: Bamboo 5 has landed! Take the tour here.
*I’m talking small “c” communism as theorized in 1848, not the shenanigans that actually manifested 70 years later.