CI/CD2 & Continues Debment: How to deal with technical debt

Originally written at pooyan.info

Who is the author? Check out my profile on LinkedIn.

Last week, I read this article by Marc H. Weine on Medium about technical debt and how to deal with it. I liked many points in there and want to add some more thoughts to it.

By Marc H. Weiner

As a person who has worked in software development for about 20 years in different companies, as an employee or consultant, team member, or team leader, I have seen many projects that have suffered from technical debt. This usually happens when speed overtakes quality, and the team is pressured to deliver features quickly, mainly because of all the misunderstandings around Scrum and Lean methodologies. That's one of the reasons why most startups fail. (of course, there are many other reasons, like delivering too slowly, building something no one wants, etc.)

If you are a software developer, team leader, or manager, ...

Remember:

Pooyan Razian

In the past few years, in the projects that I had the opportunity to lead, even though my authority was relatively limited, I tried to influence a culture that minimized this problem. Shall we call it "CI/CD2"?

Let's first see what technical debt is. Then, we will see how to deal with it. And finally, we will explore what "Continues Debment" is.

Spoiler alert: CI/CD2 is not a tool or an Ops thingy. ;)

Main types of technical debt

• Code debt:
  
  Poorly written code that is hard to understand and maintain.

  • Testing debt: When managers don't see testing as a part of software development. (and I have seen that a lot even in big companies!)
  • Architecture debt: Outdated designs that can’t scale, or even worse, no architectural thoughts at all! (I have seen this a lot in the Nordics even in highly regulated industries!)
  • Tooling debt: Outmoded or fragmented tools that slow developers.
  • Inefficient process debt: Repetitive manual processes that could be automated. (manual testing, manual deployment, etc.)
  • Knowledge debt: Lack of documentation and onboarding processes. Lack of the business why's behind things. # Why do we see technical debt everywhere?

• Misunderstandings around "Scrum"
  
  
  While the primary goals of Scrum were to promote effective communication (not over communication or isolated islands), adaptivity, efficient documentation (not over documentation or not documenting at all), open-mindedness, and continuous thinking in software development, it has been misunderstood by many who made their own versions of it to advocate their traditional project management practices behind this fancier new name, "Scrum". What we have forgotten is the Agile Manifesto, which wanted to promote better ways of developing software by doing it and helping others do it through these values:

• Individuals and interactions over processes and tools.

• Working software over comprehensive documentation.

• Customer collaboration over contract negotiation.

• Responding to change over (blindly) following a plan.
  
  
  (of course, teams can have their own values, but they should be specified)

But we ended up valuing complex processes and endless meetings over what matters the most. This led to a culture where we kill time and creativity and instead promote "report to me every day" in the name of "Standup meetings"! I bet if you ask Scrum masters, 7+ out of 10 will tell you the main reason for the daily Scrum meetings is to either report to the Scrum master or to answer the 3 questions, which is a nice name for the same "report to me" thing.

Source

• Short-term thinking Also, with the Lean approach, we wanted to quickly evaluate our assumptions and adapt to what we learned from those studies. But we ended up thinking Lean, Scrum, Agile, all mean "fast fast fast"! I even remember a newly-became-Scrum-Master once told me Agile means fast, so we should always push our limits to deliver faster and faster and faster, no matter what!

Source unknown

• Overemphasis on features
  
  We have been so focused on "delivering new features" to keep our higher-level managers happy that we have forgotten the importance of building what is needed the most with acceptable quality, code health, and long-term sustainability. Also, this leads to building complex "everything solution" that no end user is willing to interact with.

  • Not seeing testing as a part of development The majority of projects that I was added to either had zero test coverage or just something to show a green indicator in the CI/CD piepline saying "tests are passing", while by tests they just meant 1-2 unit tests. Also, remember, testing is not just one thing. We have various types of testing which I explained in another article.
  • Not seeing refactoring as a part of development Technology constantly evolves. The libraries and tools we use, coding standards, and architectural best practices (based on what stage we are at, cost, technology limits, etc.). Also, we should learn from our mistakes and (hopefully) become better developers day by day. We can either let all these learnings go to waste or apply the related ones to our codebase.
  • Lack of standards When you have a team of 10 developers, you have 10 different ways of thinking and coding. Diversity in our thoughts is a good thing. But, when it comes to understanding the codebase, if we do not accumulate the team's knowledge into one package, our products will look like a patchwork quilt! If I look at your team's codebase, it should feel as it is written by one person (one team as a whole), with similar naming conventions, styles, patterns, etc. no matter how experienced each developer is.
  • Not seeing documentation as a part of development Documentation is not a luxury, extra nice-to-have thing. Similar to testing, it is a part of what we deliver. Shall we write a book for every single feature that we add? Definitely not! And it is crucial to efficient documentation that adds a lot of information with the least possible words. (based on information theory)
  • Business and dev separation This usually happens when the culture says whatever a manager says should be treated as a god's word without needing to think and understand the crucial questions like who needs it? Why? and what is our suggestion for making it happen? and why this is a good solution? etc. While this is not a direct reason for technical debt, it is a reason for not prioritizing the right things based on the combination of business needs and technical limitations/possibilities/risks.
  • Lack of trust or management obsession Business is mainly about money, cash flow, and return on investment (ROI) which is usually calculated by feasible things like money in money out. When you spend money, you expect more of something to happen, right? So, more "hours of work", more "features", more "reports", more "meetings", etc. But that's not how creative jobs work, for people who should use their brains to solve problems. Also, the word "master" in Scrum Master has made many SMs think developers are their slaves who should not be trusted. As a result, we have this "report to me every day" culture in many places, done in the name of daily standups. While the primary goal of those daily meetings came from XP (Extreme Programming), not Scrum itself, it was to let the team autonomously plan their day without needing a "daddy SM" or a "master" to check & push them beyond logical limits. While this is not a direct reason for technical debt, it can lead to wrong prioritization and a culture where things are done just to please the manager (or SM), not to solve customers' problems and add value to the business.

source

• Debt is seen as separate:
  
  Teams treat debt as an isolated task rather than part of daily development.

  • Reactive mindset: Problems are fixed only when they break production.
  • No ownership: Nobody is accountable for maintaining code health. Or worse, one person, usually the poor junior, will be assigned to do "firefighting" when things really go wrong. # The real cost of ignoring technical debt

Source: Vincentdnl

When technical debt accumulates:

• Projects slow to a crawl. You fix a bug here and create ten more somewhere else.
• Slowness, when people already know your brand = less innovation = less competitive advantage = less revenue or even losing the market. (A bad "Scrum" done by inexperienced managers, instead of making you more Agile, can kill your agility when you need it the most!)
• Simple changes become risky and time-consuming.
• You might reach a breaking point where rebuilding from scratch becomes the only option.
• People burn out trying to maintain unmanageable systems.
• You will lose key developers.
• No one is willing to join your company or stay there.

Source

And without innovation, reliable products, a poor user experience, you will:

• Lose your reputation.
• Lose your customers.
• And then even lose your whole business!

source

What are our options?

Cleanup Sprints

A common approach is to schedule a 'cleanup sprint', hoping a one-off effort will fix years of debt. However, these short bursts rarely address the root problem and typically only delay the inevitable collapse. Sur, this can save mid-level managers and Scrum masters for a while before they are fired! But putting a band-aid on a broken leg is not a solution.

Sunsetting

Another approach is to sunset the project and start from scratch. This is a last resort to save the business. The cost of rebuilding is usually high because, with the same culture/people, you will end up building the same thing again! With new people, you will have to pay the price of onboarding and knowledge transfer, which usually never happens. If you have a non-maintainable product, that means sharing knowledge is not a part of your culture, and you cannot change people's habits/personalities overnight!

Incremental Refactoring

The best approach is to refactor incrementally. This means addressing debt as part of your daily workflow. Similar to the fact that testing is a part of development, refactoring should be too.

Continuous Debment

Debment = Debt Management

CD2: Continues Debment + Deployment

Inspired by CI/CD, Continuous Debment (CD2 = Continues Debment + Deployment) integrates debt management into our daily tech processes:

• Connect business and customers with the tech lead: Ensure the responsible person (tech lead in this case) assures everyone understands the why's behind the business needs, features, and the codebase.
• Avoid the "manager is a god" culture: Trust your team and let them understand and plan for themselves.
• R&D mindset: Encourage your team to learn, play with the codebase, and apply what is needed. Make sure they have time for that. Timebox it if needed.
• Team Ownership: Make debt everyone’s responsibility. Empower your tech leads to spend time on refactoring and code health. Guide them if needed, and provide them with the context and even extra temporary consultancy help if they need it.
• Incremental Refactoring: Address debt as part of your daily workflow.
• Proactive Debt Reviews: Treat code health as a first-class citizen. Coding standards, automated tests, automated test coverage checks, code reviews, and pair programming can help.
• Small, Frequent Refactors: Address issues before they snowball.
• Automated Checks: Integrate tools to check for test coverage, using old libraries, etc. Even though, as of now, you cannot automate everything and measure technical debt; you can start with some automated checks alongside your existing continuous manual checks added to your process. # Limitations

The main problem is that technical debt cannot be measured accurately. Our brain understands mathematical models better than abstract concepts, and that's why understanding quantitative metrics is easier than qualitative ones like "better code quality", "better architecture", etc.

However, we can measure the effects of technical debt with things like:

• Time to deliver a new feature.
• Time to fix a bug.
• Time to onboard a new developer.
• System performance.
• Downtime.
• Security incidents.
• Customer complaints.
• Customer satisfaction.
• Employee satisfaction.
• Employee turnover.

Bear in mind that counting the number of delivered features or fixed bugs leads to incorrect conclusions because the complexity of those features and bugs hide behind those numbers and can vary a lot. We cannot even count based on the delivered "story points", because "what does it mean to say 1 story point?" I bet if you ask that from 10 people, you will get 10 different answers!

Conclusion

Technical debt is a natural part of software development. But when ignored, it can cripple your project. Software development is all about balance. We are not supposed to build everything from day 1, and we can skip nice-to-have for the future. What is "nice to have"? That's a crucial thing when it comes to technical debt. Things like automated testing, documentation, and architectural decisions are essential parts of the process for long-term sustainability.

To manage debt effectively:

• Trust your team and empower your tech leads.
• Addressing tech debt should be a part of development.
• See documentation as a part of development.
• See refactoring as a part of development.
• See testing as a part of development.
• See coding standards as a part of development.
• See architectural decisions as a part of development.
• Understanding the business needs and the why's behind them is a part of development.
• Continuously evaluate and treat debt like a first-class citizen.
• Encourage your tech leads to make tech debts everyone’s responsibility.
• Add automated checks for repetitive tasks where possible.
• Write down and enforce coding standards, security standards, ways of working, etc.

By continuously addressing technical debt, you can keep your project healthy and sustainable.

And remember, CI/CD2 is not a tool or an Ops thingy. It's a mindset that integrates tech debt management into your team's day-to-day job. Also, it is not easy to measure technical debt because of its qualitative nature. However, you can measure some of the effects of it and use those metrics to roughly understand the health of the project.

What are your thoughts on technical debt?

Do you have any experience with it?

What are your strategies for managing it?

Let me know.

---

If you liked the article and want to keep me motivated to provide more content, you can share this article with your friends and colleagues and follow me here on Medium or LinkedIn.

Copyright & Disclaimer

• All content provided on this article is for informational and educational purposes only. The author makes no representations as to the accuracy or completeness of any information on this site or found by following any link on this site.
• All the content is copyrighted, except the assets and content I have referenced to other people's work, and may not be reproduced on other websites, blogs, or social media. You are not allowed to reproduce, summarize to create derivative work, or use any content from this website under your name. This includes creating a similar article or summary based on AI/GenAI. For educational purposes, you may refer to parts of the content, and only refer, but you must provide a link back to the original article on this website. This is allowed only if your content is less than 10% similar to the original article.
• While every care has been taken to ensure the accuracy of the content of this website, I make no representation as to the accuracy, correctness, or fitness for any purpose of the site content, nor do I accept any liability for loss or damage (including consequential loss or damage), however, caused, which may be incurred by any person or organization from reliance on or use of information on this site.
• The contents of this article should not be construed as legal advice.
• Opinions are my own and not the views of my employer.
• English is not my mother-tongue language, so even though I try my best to express myself correctly, there might be a chance of miscommunication.
• Links or references to other websites, including the use of information from 3rd-parties, are provided for the benefit of people who use this website. I am not responsible for the accuracy of the content on the websites that I have put a link to and I do not endorse any of those organizations or their contents.
• If you have any queries or if you believe any information on this article is inaccurate, or if you think any of the assets used in this article are in violation of copyright, please contact me and let me know.