Introduction
For some organisations, working in a DevOps model really means “We have no operations capability and rely on the developers to build, deploy and manage all of the environments from development to test to production. Mostly by hand. Badly.”
This is quite frustrating for anyone from an operations background.
Operations is a discipline, with its own patterns and practices, methodologies, models, tools and technologies. Just because modern cloud hosting makes it easier for developers to deploy servers without having to know one end of a SCSI cable from another doesn’t mean they know how to do operations.
DevOps means Development and Operations working together collaboratively. It’s about putting the operations requirements surrounding stability, reliability and performance into development practices, whilst using developers in the management of the production environment (e.g. having them on-call, or leveraging their skills to help automate key processes).
This is quite different to the laissez-faire ‘anything goes’ model where developers have unfettered access to the production environment 24x7x365, changing things as and when they like.
Change control was invented for a reason, and whilst it has becomes its own cottage industry involving ever more bureaucratic layers of ‘management by form filling’, the basic discipline remains sound. Think about what you want to change, automate it if you can, test it, understand what to do if it screws up (roll back plan), document the change, make sure everyone knows when, where and how you are making the change, and make sure the business owner approves.
An Operations Manager’s Perspective:
When I took over the operations of a high-volume UK website, I spent the first three weekends working; fighting fires and troubleshooting production issues.
My first action after that baptism by fire was to revoke access to production for all developers (over howls of protests). Availability and stability immediately went up. Deafening silence from development, invitations to beers from the business owners.
Next step was to hire a build manager to take over the build and deployment automation, and a release manager to coordinate with the business what was going into each release, and when. End result – 99.98% availability, with more releases, deployed within business hours without impacting the users, and a lower TCO. The business was much happier, and so was the development manager, as he was losing far fewer developer-hours to fire-fighting production issues, and hence the overall development velocity improved considerably. Win-Win.
Was that a DevOps anti-pattern? Did I create more silos? Probably… but in a fire-fight a battlefield commander doesn’t sit everyone down for a sharing circle on how they are going to address the mutual challenge of killing the other guy before he kills you. Sometimes a command and control model is the right one for the challenge you face (like getting some supressing fire on the target while you radio in for some air support or artillery!).
That said, once we’d developed a measure of stability we did move partway to a more DevOps pattern – we had developers on-call 24×7 as third line support, we virtualised our environment(s) and gave developers more control over them, and we increased our use of automation.
Organisationally we remained siloed however – we were incentivised in different ways (operations emphasising availability, development emphasising feature delivery). We remained in essentially a waterfall delivery model and Ops vs Dev was a constant struggle for manpower and resources. All the usual problems that the DevOps movement is trying to address.
DevOps involves development and operations working together. It’s about synthesising a better, more agile and cloud-oriented way of working based on the best of both disciplines. Anything else devalues the DevOps concept.
