Tuesday, December 13, 2011

Technical Debt - from Owing to Owning

The term "Technical Debt" is used often in Scrum and otherwise Agile discussions. We know - or at least assume - it's a bad thing to have. Some best practices concerning definition of done for a story is to not leave any technical debt laying around. But what exactly is this technical debt all about? How does a team manage it? is it necessarily bad and should it always be avoided - at all costs?

Defining Technical Debt
Technical debt is any engineering activity that a Scrum or Agile team //should have done// but for whatever reason hadn't yet.

Some common examples might include:

  • Smelly code that needs refactoring
  • A broken automated build-server which while broken forces the team to do manual builds
  • An automated acceptance test which is constantly and falsely red - not because there is a bug but because the test itself isn't properly testing the right outcome (a.k.a False Negative)
  • A piece of code which doesn't have any tests at all

It's obvious from these examples that the team needs to do some work to get the debt "paid" - refactor the smelly code, fix the build-server, fix the test or write the missing test.

Work to do or Debt? What's the difference?
When I go to a store to buy groceries, I will take into account that whatever I'll buy will cost me money. - everything has a cost to it. Before I buy anything I don't owe anything, but once I collect my groceries and go to the cashier, I'm expected by mutual agreement and law to pay for what I've taken. But sometimes I find I forgot my wallet at home - and the shop owner might agree to give me the groceries and have me owe him money - that turns the cost into debt.

So back to software - when the Product Owner requests a story to be coded, during sprint planning the team will estimate and plan the total cost of the work - they will write down all the technical tasks that need to be done in order for the story to be considered "Done", and throughout the sprint will do everything in their power to complete all those tasks.
At this point the team has agreed - and committed - to do all the tasks. However sometimes (and sometimes more often than not) the team might find in the end of the sprint or even way after the sprint had ended that there are some "undone" tasks - things than still need to be done even though they should have been done before. This is where we view these tasks as not only "TODO Tasks" but as "Debt". The team owes the tasks to the PO or to themselves by power of their previous agreement and commitment.

Who do we Owe this to?
In Scrum the agreement on what tasks should be done for completing a given story are derived from the "Definition Of Done" (DoD). The DoD is an agreed upon list of criteria defining what it means for a story to be done - for instance it might include "there must be automated tests for all functionality", "all code should be refactored so there are no code smells left" and "All bugs should be fixed".
So if a team has "Technical Debt" - it's likely to be a violation of the DoD.
But the DoD itself is formulated through mutual agreement between the PO and the team, and will likely include both items that are important to the PO (like not having any bugs) and items that are important to the team itself (like having clean code or automated tests, since these will help the team move forward faster and make life easier for them down the road).
So any violation of the DoD is not a violation of a commitment to the PO, but in fact a violation of commitment to the team as well.
Owing technical debt is owing something to both the PO and to the team itself.

Don't get into debt in the first place? I wish...
One of the most common pieces of advice concerning technical debt is to not have it in the first place - I've heard consultants say that the PO should not accept stories for which all tasks have not been done yet, or that the team should not "claim points" for undone work. At some point the Scrum Guide included some pages dealing with "Undone Work" in it (I recall they have been removed in the latest edition, but don't take my word for it).

But in the end - there will always be "undone work" discovered somewhere - maybe at the end of the sprint the PO really needed that feature delivered to gain a business advantage and the team came to agreement with the PO that the code would not be fully refactored.
Maybe the team could get away with manual builds or manual tests for a sprint or two and investing in automation wasn't justifiable at the time, although now they realize they would be better off automating things.

Either way - technical debt (like financial debt) is usually brought to being through some agreement, done in good faith and for good reasoning at the time it was agreed upon.

Types of Technical Debt
I find most "Technical Debt" can be categorized in to one of three types:

1. Debt:
This is the work that we knew we needed to do but for some reason decided not to.

For instance, if I gave a dinner party at my home and decided not to wash the dishes - obviously I'll have to wash them the next morning.

In software, this might include:

  • If a team had committed to writing automated tests but decided with the PO to not invest in them for the current sprint - they will probably need to catch up at some later time.

We call this Debt because the team had initially committed to do it but has later decided to postpone.

2. Discovered Deficit:
This is work we discover needs to be done only afterwards.

In my kitchen I might some day discover I have leaky plumbing. I didn't know of this ahead of time and had no way of foreseeing it, and will now have to take care of the situation before my next dinner party.

In software we often discover "Technical Debt" in hindsight - for example:

  • While working on one part of the system we did refactor the code the best we could, but now - several sprints later - we have learned new coding patterns and realize that what was previously considered good code might be better if a new pattern was applied, and we want to refactor that code.
  • We might realize we've been writing automated acceptance tests but in a form which is now very hard to maintain, and would be better off reorganizing those tests so they will be easier to maintain and evolve further.
  • As the system runs throughout time in production (with real users) a database table might fill up beyond our initial expectations, and we realize that in order to meet our performance needs we should normalize it separating it into two separate tables (this is is fact a form of refactoring).

We call this Discovered Deficit because even though the team did everything they had committed to at the time, only with their newly gained knowledge, insights and perspectives can they now see and plan what they might have done better.

3. Technical Investment
Back in my kitchen - I wash the dishes, I have no plumbing problems, and the dinner parties are a great success - so I decide to buy a dish-washer to make my life easier.

In software, we might see opportunities to make the engineering work easier or better:

  • We might want to install, learn and use a new testing framework, or better build-server 
  • We might want to learn new techniques of working. 
These will presumably be good investments and although they will take some time (and maybe money) - they are assumed to pay off by improving quality or team velocity and so are good for business. Hence the term Technical Investment.

Is it bad?
Debt has a bad connotation - nobody likes being in debt, and if we are - we probably did something wrong. Once we realize that the technical debt was in fact agreed upon and taken into account - we can stop seeing it as a negative thing and start viewing it objectively - as mere "work to do".
Even calling this "undone work" smells bad - if it's "undone" it means somebody should have done it but didn't.
Looking at the categories mentioned above - we can see that much of the "Technical Debt" might in fact be the fruit of new insights or simply a worthwhile investment opportunity - not debt at all.

Seeking a more objective term - we might call these "Technical Tasks", or "Technical Stories", although these terms have their own baggage among Agile circles.

However we name it - this work differs from normal story tasks in that "normal" tasks (planned upon in the sprint planning meeting) are commissioned by the Product Owner ahead of time and are expected to create immediate value to the product - the PO realizes the tasks will need to take place in order to fulfill a story he/she wants done. The PO takes responsibility for the cost of these tasks as part of the responsibility for the product backlog.

The "Technical Stories" on the other hand are not going to be commissioned by the Product Owner anymore - the PO won't take responsibility for them, and won't prioritize them for upcoming sprints, since they don't drive immediate value to either user or customer. This means someone else needs to take responsibility for them - probably the team itself. They should be the ones wanting it to be done and pushing for it to be included in a sprint.

Responsibility and Ownership
Responsibility is a key factor to Agile software development - we want self organizing, responsible teams and individuals acting proactively and with respect to each other.
In Scrum the responsibilities are clearly divided - the Product Owner is responsible for the product while the Team is responsible for delivering it. This means the team is indirectly responsible for the quality of the code, and as a self-organizing and constantly improving team also responsible for it's own quality of work, it's pace, focus and capabilities.

Seeing that the team is responsible for their quality and performance, combined with the above-mentioned insight that the team in fact owes the "Debt" to themselves, turns the team into yet another stakeholder for the PO - a stakeholder that pushes to include "paying off technical debt" into each sprint.

So if the Product Owner //owns// the product, who //owns// the technical debt?
The team of course. They are owners of these "Technical Stories" just as the PO owns the "user stories". They own the list of "Technical Stories" (or "Technical Backlog") just like the PO owns the product backlog.

This puts a whole new perspective on what technical debt is - almost like the saying goes "it's not a bug - it's a feature", but in this case "it's not technical debt - it's a technical story".

From "Team" to "Code Owner"
Ownership of the product by a single responsible PO is what drives the Scrum process into creating working software that creates value to the business. Strong product ownership is based on drive, vision, a clear understanding of the product and business needs and the authority and capability to make decisions concerning prioritization and return on investment (ROI) of each feature or story.

Likewise, strong ownership of the "Technical Stories" should be driven by the same traits - a team should have firm belief in the need and in the gained value of completing each "Technical Story", should be capable of decision-making concerning which "Technical Stories" to do and when they should be done, and should be empowered to make those decisions - by mutual agreement and respect from the PO.

In this perspective the team is not only responsible for delivering working high quality software, but becomes the "Code Owner", equivalent to the "Product Owner" in the essence of ownership, while differing from the PO in the subject matter that is owned.

Viewing "Technical Stories" from this perspective enables a much more responsible, strong, proactive and positive approach to dealing with the tasks in hand than with the negative perspective of "Technical Debt". And this is the main motivator - not dealing with "why it happened", but rather with "how can we make things better".

Coming next.. "It's ours! Now what do we do with it?"
Once the team realizes that technical debt is not a bad thing but in fact something they own and are responsible for - they need tools and techniques for managing it.

I'll be covering that in an upcoming post. It's next in my backlog...

1 comment:

  1. Just wanted to share a conversation sparked by feedback from André Brissette on the LinkedIn
    Agile Coaching group

    Thanks André!

    On 12/17/11 8:51 AM, André Brissette wrote:
    Hi Avi, here is some feedbacks

    Your article is too long. many elements are very valuable in your article. But in my opinion people are used to (or only have time for) read smaller article. What you could do is to break it in multiple article with title like:

    1) Owing technical debt is owing something to both the PO and to the team itself. (I love this one :-)
    2) Type of technical debt
    3) Responsibility and Ownership of the debt
    4) etc..

    Another reason for breaking it down in multiple articles is to facilitate conversation with your readers. For example, in the section "Type of debt" they were many points that I wanted to comment or argue. But by the end of the article, my focus was distracted by other (interesting) elements. I think people are more likely to comment on shorter article.

    by the way, it looks you have a typo here "In this perspective the tam is not only responsible for ..."

    It is a very interesting article and it looks like you know your subject well. good work!

    have a nice day,

    On 12/17/11 11:18 AM, Avi Naparstek wrote:
    Hi André, and thanks for the constructive and honest feedback!

    I see what you mean by it being too long.. And you make excellent reasons for breaking it down.

    To be honest I was rather eager to get it out - I've just published it in some more forums so I'm not sure I'll be re-editing it, but will be sure to take that in mind for the following posts. If it's OK with you I'll want to credit you for giving me that feedback.

    I guess it's also a matter of media as you pointed out - in a magazine or academic publication a long article might be appropriate, whereas in a blog-post published in forums for discussion a shorter one would be more effective. I definitely owe you one for giving me that observation :)

    ...And you are right about the typo of course, I just ran a spell-check and found some more.. Yikes!

    So thanks again, I would be interested nevertheless in hearing your thoughts, comments and arguments in the group. Maybe a good discussion will come out of it after all.

    Best wishes,

    On 12/19/11 6:45 AM, André Brissette wrote:
    Hi Avi,

    Of course credit is always welcome :-) If you want we can put this exchange public. I initially choose to make it private as my comments were more on the form than the content.

    I checked the thread this morning and it looks that you spark an interesting discussion,
    thank for the initiative!

    Happy holidays to you and your family,

    On 12/19/11 Avi Naparstek wrote:

    I just skimmed through the conversations - I was glad to see you participated. I've yet to have a deeper look at what was written.

    And I am actually glad you commented about the form - it's just as valuable and is a rarer kind o feedback to get.

    I appreciate the fact that you feedbacked privately, although with some courage.. hmm.. . scrum? courage? why not? here goes...

    Happy holidays to you too and to everyone how will be reading this!