Introduction
Many organisations uncover mountains of technical debt during their migration to the cloud. But what is technical debt and how can DevOps help us deal with it? In this post, we discuss five ways DevOps can turn your technical debt mountain into a molehill!
What is Technical Debt?
Technical debt is the accumulation of sub-optimal technical decisions made over the lifetime of an application. Eventually, it gets harder and harder to change things: it’s the ‘sand in the gears’ that sees IT initiatives grind to a halt.
For example, poor state management in an application can make a horizontal scaling strategy harder to implement. Before you can do what you really want to do (horizontally scale your application so you can deal with increasing traffic) you need to re-write the state management portion of the code. The effort taken to ‘do what’s needed before you can do what you want’ is technical debt.
It is worth pointing out that technical debt doesn’t only occur in development, it can occur in operations too. Still running an out-of-date OS that’s no longer supported, like Windows Server 2008, or Ubuntu 11.04? That’s technical debt. Not keeping your servers patched and up to date, leaving you vulnerable to cyber-attack and ransomware? Yup, technical debt.
Is Technical Debt Always Bad?
It depends. Martin Fowler’s technical debt quadrant makes the point that sometimes technical debt is inadvertent. You didn’t know what you didn’t know, but now you do, so you can fix it.
Prudent, deliberate technical debt is at the heart of Eric Ries’ ‘Build-Measure-Learn’ cycle from The Lean Startup. Sometimes, the only way to learn whether you have a viable product is to launch it and get it in the hands of customers. That may mean that you ‘cut corners’ and thus incur technical debt.
So, how can DevOps address the technical debt challenge?
1. Create DevOps Product Teams
One of the key tenets of DevOps is creating small, multidisciplinary teams (i.e. Dev + Ops) who own the entire lifecycle of a product or service from ‘inception to retirement’. Because these teams (including their product managers) feel the impact of technical debt every day, they are highly motivated to pay it back to make life easier.
Simple things can help teams tackle technical debt. The first step is to assess and track the level of technical debt in your products.
You can do this by labelling work items (in Jira, Azure DevOps or Github, for example) as ‘TechDebt’. Next, you need to carve out part of every sprint to work on technical debt items. We’d suggest at least 20% of a sprint commitment is a good place to start.
Then, within every team retro, do a quick thumbs up/down (or planning poker) on current levels of technical debt. If you get too many ‘thumbs down’ (unacceptable levels) prioritise even more repayment work in the next sprint. Keep going until your team is happy that technical debt isn’t a blocker to achieving your product goals.
Lastly, try to avoid creating more technical debt in new product development. Use the technical debt quadrant to assess design choices and minimise new debt. If you need to create prudent, deliberate technical debt, track it and allocate effort to pay it back in the future.
2. Build Shared Self-Service Platforms Via InnerSource
Increasingly, organisations are building shared self-service platforms to support product teams. These platforms help avoid the technical debt of having to build and maintain toolchains. Teams can consume a self-service platform (i.e. an internal or third party software-as-a-service (SaaS) style solution) instead. Platform teams build these services to meet the needs of product teams, along with best-practice templates, reference architectures and other code building blocks that might be needed.
Successful self-service platforms avoid ‘us and them’ silos between platform builders and platform consumers. Platforms need to be a collaborative effort. The best way to achieve this is to use lessons learnt from large, distributed open-source projects – a practice often referred to as InnerSource.
About InnerSource
The InnerSource Commons definition says InnerSource “takes the lessons learned from developing open source software and applies them to the way companies develop software internally”.
O’Reilly expands on this in its Adopting InnerSource guide:
“InnerSource represents a collaborative and empowering way of involving employees in making and implementing decisions throughout the corporation. It embodies a philosophy of human relations, an approach to rewards and motivations, and a loose, adaptable set of tools and practices.”
Platform teams might be the ‘maintainers’ or ‘core committers’ of their platform service. However, like the maintainers of any open source project, they need to be open to contributions. Code repositories must be available to everyone inside the organisation along with contributor guidelines. Contributions usually involve a ‘pull request’, which is reviewed by maintainers and merged into the platform codebase if approved.
Having an open, collaborative mindset is essential. It’s about encouraging and rewarding contributions for the betterment of all, while keeping technical debt to a minimum.
3. Use DevOps Automation to Repay Tech Debt
Inherent in any DevOps approach is the heavy use of automation. In fact, the most common DevOps model – CALMS Culture-Automation-Lean-Measurement-Sharing – has automation at its heart. Many of the platforms discussed earlier would be built and managed using automation. What’s more, the services platforms provide are often automation toolchains for the product teams to use.
But how does DevOps automation help repay technical debt?
Let’s take environment management and consider how infrastructure-as-code and configuration-as-code can help repay technical debt while avoiding future debt.
Inconsistent aka ‘snowflake’ environments are a prevalent form of technical debt that almost everyone will have experienced. When your application works in Dev but not in Pre-Production or, even worse, not in Production, the root cause is often inconsistency. This might be found in the operating system configuration, application dependencies, the myriad configuration settings upon which the application depends or the communication between different services that comprise the application. The time you spend troubleshooting issues like these is the cost, the impact, of your technical debt.
Infrastructure-as-code and configuration-as-code enable you to precisely express your intent for how you want the environment to be. Then, via the power of tools like Terraform and Puppet, you can (declaratively) ‘make it so’, consistently, in every instance of that environment. Containerisation, which we’ll talk about later, takes this to the next level.
The time you spend NOT troubleshooting deployments due to snowflake environment issues is time you can invest paying back even more technical debt. This leads to a ‘virtuous circle’ of automation.
‘As-code’ automation also has another benefit. Holding DevOps automation as code, in a repository like GitHub, rather than describing config settings in verbose Word documents or Visio diagrams, makes it simpler and easier to iterate and improve. People can discover the code easily via search, submit suggested changes via pull requests, or even fork code, modifying and extending it to meet their needs. This ability to iterate means automation code is less likely to get ‘out of date’ thereby avoiding another form of technical debt.
4. Use Containers to Simplify App Deployment and Management
What are containers and how can they help you repay technical debt?
In short, containers enable you to hold your application, its configuration and OS dependencies in one lightweight bundle that’s easier to deploy and faster to provision.
As with the environment management automation example discussed earlier, the portability of containers simplifies everything. Container orchestration tools like Kubernetes go even further, enabling automation of the container lifecycle in production, freeing DevOps teams to focus on higher-value tasks (like re-architecting applications to reduce technical debt).
Organisations building new, cloud-native applications typically use containers to improve flexibility and scalability at a lower total cost of ownership (TCO). The ‘microservices’ architectural pattern works well with containers and, more importantly, the small multidisciplinary DevOps product teams mentioned earlier. Each microservice is small, with a clear bounded context, and managed by a long-lived team. So, technical debt is easier to spot and fix. A win-win scenario for everyone.
5. Using DevOps to Build an API-Centric Model
The microservices model discussed above is one way to implement an API-centric application strategy. Jeff Bezos famously challenged his teams to ONLY use APIs to communicate between their systems “or be fired”.
Your organisation might not be quite so draconian. Nevertheless, encouraging teams to build and consume APIs with well-defined, versioned interfaces is a great way to reduce technical debt. Often, technical debt is caused by different systems accessing services and data in ways that other teams don’t expect. For example, TeamA directly reads data from a table created and managed by TeamB. If TeamB changes the schema of that table to meet their needs, they might inadvertently break TeamA’s application because of this hidden dependency.
APIs make these interfaces more visible and less fragile. Each product needs to adhere to a published API, with clear future roadmaps, ideally supporting ‘semantic versioning’. If at some later stage they need to introduce a ‘breaking change’ to the API specification they can release a new major version of the API, with support for the “old” API for some pre-determined support period. All this adds up to less fragility = less technical debt.
DevOps patterns and practices, like using CI/CD code pipelines to build, test, package and deploy applications, enables teams to move faster, with less risk and less technical debt.
Code pipelines also provide end-to-end traceability of application code – from user story to code commit to released package. This makes it easier to manage technical debt. If you’ve tracked and labelled technical debt as discussed earlier, you can see the impact it’s having in live systems and on your customers. This helps derive clear, customer-driven priorities for where to focus technical debt repayment efforts.
What next?
Hopefully, these five ways will help you tackle your technical debt challenge with DevOps. If you need any help our talented cloud engineers would love to get involved!
