My take on TDD and Big Data

Following a discussion with Gil Zilberfeld on TDD and Big Data during the recent APIL14 conference, Gil sent me this video:

Admittedly I have approx 10 years of experience with TDD and BDD, but have never had the chance to try TDD with Big Data. It sounds challenging and interesting.

My own take on this is similar to what the video described - I believe in BDD (Behavior Driven Development) and that seems to fit nicely in the Big Data world.
1. Define your expectations using a scenario (with Given-When-Then language or otherwise define the expected behavior of the system or a subset of it)
2. See the test fail
3. Build the system (data pipelines, filters etc) so it makes the scenario pass
4. Refactor, rinse and repeat...

Here is a contrived pseudo-code of what I have in mind:
Suppose I am google (hey, why not?) and I want to build a search engine (how come they never thought of it?)
The search engine would have a crawler that crawls websites and streams data into a distributed big-data index.
People at home would enter a search query and get results from the index. Obviously the results might vary depending on how close the person is to a data node, and on if the data has synced between nodes yet.

So... ready? (I know I'm not but what the heck, it's only the whole internet reading this)

Scenario 1: Indexing a site into one node, and searching from another node before data has synced

Given two nodes
and given a website called "Agile Shmagile"
and given that the site is only indexed in node1
and given a user that is close to node2

When  the user searches "Agile Shmagile"
Then she should not get the website in the search results

Scenario 2: The site get's synced into node 2 - and now the user should get the website in the search results

Given all of the above from Scenario1
and given that the website has synced into node2

When the user searches "Agile Shmagile"
then she should get the website in the first place in google
and "Agile Shmagile" authors should get a trillion bucks
and they should retire to the Bahamas....

Hmmm.... I think it's good to go live - I just need to make that last assertion pass...
Bahhh - should be easy.

:)

Would you believe it? Metrics really **CAN** be gamed!

Recap of the Ready-Steady-Sprint! session at the APIL14 conference

This week I had the pleasure and honor to attend the Agile Practitioners IL 2014 conference - not only as an attendant but as a speaker for the first time.

Together with Shirly Ronen Harel and Dan Kuida (my partners in the AgileLife initiative and other projects, and close friends) we ran an interactive session called "Ready-Steady-Sprint!". It was a fast paced card game developed by us, which teaches first-hand what working together as a scrum team is like - both in terms of team-dynamics, pressure under a timebox, and specifically utilizing Scrum and XP practices.

So if you don't have the patience to read further I'll save you the trouble and just say 

IT WAS A BLAST!

Dan and Shirly are so proud of me...

...making fun of myself on the podium

Some boring history:
The game was originally prototyped by my eldest son (Michael) and myself. BTW he got it straight away! He understood what a bug is, what refactoring means and how important automated tests are. Go figure.

Original prototype cards - to be sold on eBay for a small fortune soon

It was later crafted into a card game (printed in a home printer on cardboard and manually cut with a knife) as a recap activity for a team I worked with - summarizing the things they had learned throughout their first scrum project.

This was their first project together as a team, and their first acquaintance with agile and scrum.
During the project I was designated to teach them Scrum and some XP practices and of course to lead the project to success...

The project was for a big government office and a public, high profile, high country-wide PR initiative.
Yikes... not something you'd like to fail in.
However - the project was a huge success (I know I know - this sounds unbelievable). The client was delighted, the project was #DELIVERED on time and on budget, it had lots and lots of scope changes throughout and we adapted to the changes and managed to deliver the right product in the designated timeframe.

Also the team itself had a chance to learn and practice some interesting Scrum and XP practices and later matured to work together on other agile projects.
Yay!

The hand-cut home-printed cards for the original project-X team

Some time later we formed the AgileLife group - composed of Agile professionals who wanted to take Agile practices and mindset outside of the software realm into real life - including arts, education, community and more - and take some real-life wisdom back into the software world as well.
We adopted the card game as one of our group projects, and I teamed up with Shirly and Dan to present the game in AgileSparks' Agile Israel 2013 conference. We submitted a call-for-papers but were rejected. Awwww...... :(

So when Ilan approached me asking if I'd like to give a talk in the APIL14 conference - I had no question about it. We would give the game as an interactive session. By hook and by crook we would.
The focus of the conference was "Learn from the best" - so I figured the best way to learn is from experience, and the best teachers are our teams. Nice twist and a great excuse to submit the game :)

Fresh from the print-shop

One of our biggest challenges here was working with a large number of participants - until now, the game had only been tried with small teams, so the entire team would play together and I would act as the facilitator of the game.

Now in the conference - we were faced with the horrifying possibility that more than 10 people might attend... So after overcoming the initial urge to opt out and run for our lives, we decided to embrace the challenge and devised a format which would fit multiple teams - actually simulating a scaled-out agile adoption. This meant having several teams each playing their game, and scrummasters doing Scrum-Of-Scrums syncing between the teams. Wow! talk about getting a chance to fail big-time :)

Another dilemma we had was how to plan for the session - we didn't really know how many people would attend. We didn't know if to plan for a scrum-of-scrums session or for a single-team session. Seeing this as any other kind of product risk in agile planning, we took Mike Cohn's advice and did both - we devised two game plans - one in case we have only a handful of attendees and another for in case we have more. We had a third plan for in case nobody attended, but that's another story altogether...

The cards we used in the conference - professionally produced for us by Dina Kuida of DDK graphics studio

Enough bullshit - let's hear the Game Plan
The game is similar to the "Race" game if you happen to know it.

The objective of the game is to complete "tasks" using "time" cards.
The players are grouped into teams (each table had a team of 7-9 people) and each team elected one member to be the scrummaster for that team.

In the beginning each player gets 5 task cards (this is their "TODO" pile) and 5 other random cards (this is their "Hand"). In the middle of the table we placed a pile of cards ("the Pile") with their backside facing up.

Each player in his or her turn takes a card from the pile and needs to use a card (either the new card or another card in their hand).
Some cards are pretty straightforward - for instance, you can use a "time" card to complete a "task" - in which case you throw both the task and the time cards into the "DONE" pile.
But some cards are more challenging - there is for instance a "bug" card - which needs to be placed on the TODO pile and later handled using another time card.
Or there is a "Code Smell" card - if you get it you need to place it on your TODO pile and from that moment on - each task will take more time (=more time cards) to complete, until the code smell is taken cared of - using another "Refactoring" card.
Also there are special "Protection" cards - which once used, protect the player from various problems. For instance, if you use an "Automated Tests" card - you are protected from bugs for the rest of the game.

Our mugshots in the conference PR

The teams have 25 minutes (a Pomodoro) to complete all the tasks of all the players.
In addition - In the beginning each player can only complete his or her own tasks (high truck factor).
There is a special "Pair Programming" card that enabled two players to reduce truck factor - and combine their piles into one.

Throughout the game - the scrummaster maintains a sprint burndown - counting how many tasks remaining each minute of the game - and is responsible for encouraging and helping the team succeed, as well as making sure nobody cheats.

Every 5 minutes - the scrummasters would be called to a "Scrum-of-Scrums" - gathering together to share insights and ask for help from each other on things their teams have been doing and problems their teams are facing.

After the game is done - we hold a retrospective - each player (or some players from each team) shares some of their experiences and insights from the game.
Also the scrummasters share their experiences as scrummasters.
We would also go over the burndowns of each team - figuring out what story the chart is telling us and gaining insights from it.

In the end - each player is asked to take with them one card which they especially related to - so it would remind them of things they need or want in their day-to-day work.

The timer we used during the game - was projected on a large screen
Grab it at"www.online-stopwatch.com"

Oh! Some more boring blah blah!

The game itself is rooted in the star coaching model (strengths, obstacles, resources and goals).

In this case - the goal is completing tasks,

the basic resource is time

The obstacles are bugs, code smells, lack of time, panic etc

and the solutions to the problems are refactoring, pairing etc.

The "secret ingredient" of the game is this: There is NEVER enough time to complete the tasks, especially if we spend it on "firefighting" (refactoring, fixing bugs etc).

The best strategy to win the game is through investing in protection cards (automated testing, pairing, pomodoro, planning etc) - which makes us untouchable by the problems and leaves us free time and hyperproductive to proceed with completing tasks. 

Funny, but I've never yet seen a team grok that - at least in a first game... 

The Game Reality
In the APIL14 conference we ended up having some 40-50 attendants - so we quickly switched to "scrum-of-scrum" mode.

We allotted 10 minutes for the opening (explaining the game) - but time ran out quite quickly. We didn't have time to go over all the cards - and we decided to "throw the folks into the cold water" - assigning to the scrummasters the responsibility of explaining (and figuring out) what each cards means.
We did equip the scrummasters with a "cheat-sheet" explaining the cards, and told them they can ask us if they need to (we also walked around the tables and offered insights). Also the Scrum-of-Scrum was an opportunity for each scrummaster to ask things that they did not understand yet.

"Who's got some time for me? I need time!"

The game itself was loud, messy and highly energetic - like you'd expect from an agile software project.
Some teams were quite quick to figure out what to do,
some teams got stuck in analysis-paralysis for most of the timebox,
some scrummasters decided that doing a burndown each minute was wasteful and decided to update it every 5 minutes.
One team was stuck deciding if to invest time in pair programming on a pile of smelly code, or if they should wait until the developer refactored it first. Their scrummaster was frustrated - since he thought they should do the pair programming but was unable to convince them of it.
All in all everyone was highly engaged!

Scrum-master at work with his team

Toward the end of the 25 minute timebox we found out that we somehow had magically only 7 minutes to the end of the session (damn time... can never trust it to go as planned) and the organizers threatened to throw us out. So we shortened the 25 minute bomb-timer to 2 minutes and started "pouring oil into the fire" - telling folks "Ok 2 minutes to deadline", "The offshore customer is about to enter the building" and other kinds of stressing but not really helpful remarks - of the kind you'd expect when heading a project deadline.

We then only had 3 minutes left for the retrospective - so we just asked one member of each team to tell us an insight they'd gained and invited them to take a card they related to.

Whew! done in time but not in scope. Adapted to change and had fun throughout.

Blurry but productive

My own personal retrospective:
All in all I'm glad we tried, and glad we were challenged with the Scrum-of-Scrum format - which like I mentioned, was a first time for us.
We made some mistakes and intend to learn from them for next time.

In the midst of the mess - a smile that makes it all worth while :)

To Keep:

• Do the game! We got some feedback that it was extremely helpful and educational.
• Keep it fun - that helped keep people involved and engaged.
• Invest in it - we did put a lot of time planning it through and some money in printing the cards professionally. It was worth it.

To Change:

• Plan scope according to time - if we only have 10 minutes to explain the game, I'd use a smaller subset of the cards so we have a chance to explain them all. If we have a longer session - I'd plan a longer explanation to make sure we have all the rules covered. Also make sure we have enough time for the retrospective - I'm a bit sad that part was skimmed.
• Make sure to have it properly video'd and photographed - throughout the game we tried to take photos, but most of them were blurred since we were shooting while running around from table to table and the people themselves were in constant motion. I would invest in a professional or dedicated photographer.
• Coordinate ourselves - we (Dan, Shirly and myself) never had a chance to get together and run a simulation before the session - and I must admit we paid the price. We should take a bit of our own medicine for next time and make sure we at least do a dry-run or short planning session before the next game.

Shameless plug:

If you have attended the game - I'd love to hear what you thought - and especially what you experienced in it. Any feedback is more than welcome - except bad feedback - keep that to yourself :)

BTW The game itself is designed to be customized and tailored to specific teams and needs - for instance we have a variation where we run several sprints allowing a chance to inspect and adapt between each sprint.

If you haven't attended the game and find interest in it - please be sure to contact us at agilife@googlegroups.com or by phone +972-54-5995983 (Avi) - and invite us to give a similar session in your workplace or event. I'ts sure to be a good learning experience and a fun attraction.

Thank you!

Avi - and I believe I speak in the name of Shirly and Dan as well on this :)

And just because you've read this far - here's a special bonus track - the most-embarrassing-promo-video-ever-made-for-a-geeky-software-conference. Don't let Shirly see it - she'll absolutely have me skinned, chopped up and delivered in incremental potentially-shippable bits to the sharks if she sees it:

Personalize funny videos and birthday eCards at JibJab!

In Scrum, Is the PO a toaster?

Well, look up the Scrum Guide:

Agile North Galilee meetup #1 - Minimal Viable Meetup

So we've finally had the first Agile North Galilee meetup today!


Let's start with applying some hard Scrum Values:
Openness- I'll be dead frank here - there were 3 attendees. Yes, 3.
And now that we're over that embarrasing number, let's see what the rest of the world minus 3 missed out on...


Focus - For myself I was pretty focused on having the meetup happen, this was good because.. well.. it finally happened :) For the next meet I'd like to focus on both volume (more people) and concrete value-adding content.

Courage - I had absolutely no idea what was going to happen in the meetup, nor how many people would come. I guess neither did any of the other attendees. And we came and met nontheless. that's courage to expect the unexpected for ya.

Respect - Ok I can't help being sarcastic here... remember 3 people?
Nontheless I do respect the fact that people did not come - we took a chance holding the meetup on a Friday morning - extremely nonorthodox. Also quite honestly I did not put much effort into making sure more people would come - for me this was an experiment, a sprint-zero or minimal-viable-meetup so to speak, the goal was to practice a meetup and give ourselves a chance to learn in preperation towards the next meetup which should be bigger and value-adding to the attendees. More on that later.

Commitment - The attendees drove quite long distances to come, and like I mentioned it was a Friday morning - so kudos to their commitment! it is not given for granted. I see this commitment rooted in the North Gallee's practitioners to work in general - we have people commuting long distances and pretty much willing to turn the world around in the name of commitment - to provide for our families, while holding on to our ground to keep living up here. Throughout the conversation we had I couldn't help feel the commitment for providing not only workplaces up in the North but for actually getting work done right, in high quality and personal standards.

At this point everything was still fuzzy as to how the meetup would continue...
even fuzzier than the photo. But hey - we have some good food!

So what did we do anyway?
Apparently quite a lot. We had an agenda for 5-10 people but we ditched it in favor of a free conversation, aided by a whiteboard and a cellphone for youtube (this would probably not be possible with more than 3 people... looking at the bright side).

The first half of the conversation was mainly about work oppertunities. This is a major issue up north, and obviously one of the main motivators for the group - to create and discover new and better working and business oppertunities for northern practitioners. There have been many attemps to either bring talent and funding from the Tel Aviv area up north or to provide so-called-easy commuting means for northern folks to ocommute away. Most of these attempts have failed, and my personal feeling and belief is there are wonderful, skilled folks up here already. We are sure to find new ways of creating new venues for the local talent to prosper. Grooming this talent and connecting it to worplaces is one of the goals of the North Agile Galilee group.




The second half was actually crammed with agile topics - including:

• 

  Our board at the end of the meet
  

  Automated Testing - we learned from each other about new tooling including JSystem, Fitnesse and Selenium
• How to get work DONE-DONE in a sprint?
• Hardening Sprints - good or evil?
• Scaling Scrum in a large organization
• Throwing responsibility from Dev's to QA, and back.
• Misconceptions about what is Agile? What constitutes a successful Agile adoption? (spoiler: it's *not* about taking a 1 day course and then throwing management out the window)
• Misconceptions about what is a ScrumMaster? We watched Adam Weisbart's Shit Bad ScrumMasters Say in comparison to (sadly) real-life examples of what a scummaster is *not* supposd to to or be...
• The role of Product Owner as opposed to the role of Product Manager
• How do we handle Bugs in Scrum?
• Kanban as a means of creating evolutionary change toward agility in an existing organizatin .

My own conclusions for next time:

• Keep bringing good food.
• Find a way to get meetup.com to support Rosh Pinna, or hold the meetup in a city the meetup.com supports. Either way - registration for next time will probably be a must. 
• The next meetup is already aimed at the Carmiel/Misgav area. Even though I will be sad to get farther down north from the Northern Galilee Area, I'm looking forward to cooperating with some good folks in that area. North is North.
• The misconceptions about what Agile is, what Scrum is and what is or is not possible are probably more common than I'd expected. So for the next meetup maybe it would be best to go back the basics - an introduction to Agile values - how it is about individuals and interactions, using working software as a means for tracking progress, creating collaboration with customers and responding to change.

All in all I am grateful for this experience and hopeful about creating a new and strong Northern Agile community.

** Huge thanks to MATI North Galilee for providing us with room and facilities, special thanks to Daphna who drove up especially on a Friday to make sure we have everything we need and to Ilana who helped us close up afterwards! **

How long will it take? Wrong question!

During a conversation with a client of mine we were reviewing an idea which would (hopefully) give hime a more cost-effective solution to a requirement he had than other alternatives.
He posed the follwing observation: "It still seems like it will take some time to complete, doesn't it?"

Now that observation struck me - it implicitly asks the question "How long will it take to complete?"

This question makes several assumptions:

• There will be a certain point when the product will be "complete"
• There will be a certain amount of time invested for reaching that point
• The time invested will cost money, and therefore affect the ROI

Now looking at the assumptions from an Agile/Lean perspective, let's see how they can be looked at differently:

Assumption: There will be a certain point when the product will be "complete"

WRONG!

Agile mindset has come to relize that a product will *never* be complete.
In fact, we don't want it to be complete - why?
Because the business requirements and ecosystem are bound to change throughout the lifetime of the business.

We want to create a situation where the product can change accordingly, always changing to accomodate new requirements and new business opertunities.
A product which will be "completed" will stagnate and hold the business back in the long run instead of helping the business grow and evolve.

So we should replace this assumption with a question, or two questions in fact: 

• What is the *minimal valueable next step* we can make in the evolution of the product to help it keep in sync with the *current* business requirements?

• How can we make sure we can keep making such steps, such increments, in a sustainable pace in the future?

Assumption: There will be a certain amount of time invested for reaching that point

WRONG!

The point of completion is nonexistant, so there is no real meaning in asking how much time will be invested in getting there.

Instead, seeing as we want the product to keep changing and evolving, we should assume that each small increment will involve some time investment.

And what we shuld be asking is:

• What is the least amount of time we can invest right now in making an increment which will bring us maximum value?

(we want to hold on to Paretto's law - finding the 20% of invested time which will produce an 80% increment in value)


Assumption: The time invested will cost money, and therefore affect the ROI

WRONG!

This assumption makes another assumption - that time invested would cost money and would not save us money.

Instead, consider the following:

• Who is doing the work? if it's the customer himself, he is investing time but not paying anyone for it. So he is in fact saving money

• What kind of work are we investing in? If we invest time in creating a bad solution, we might end up with a product that will cost us more in the long run (in terms of maintenance costs). If we invest our time in a "good" solution, it would mean a solution that reduces maintenance costs in the long run = saves us money.

So the question should be:

• Can we invest the alloted time in teaching the customer how to do things himself instead of invesing in "hard labor manufacturing"? 

• Can we invest the alloted time in solutions that will cost less in long term maintenance? (consider all costs involved in maintaining your product - including hsting costs, bugfixing, DBA work, website administration etc)

To Summarise:
Instead of asking "how long will it take"? consider time as having several dimentions:

• Total (calendar) time - it is neverending and our product should keep evolving forever as long as the business keeps evolving

• Invested time - Spread it. Instead of spending a big amount of time upfront, think of spending a little bit of time every now and then.

• Cost - ask yourself how much that time costs you? How can you find ways to do more yourself (which will cost you nothing) and delegate "paid work" only for things you absolutely cannot do or for learnign how to do new things yourself

• Savings - ask yourself can you invest  time in creating a solution that will reduce your long-term costs?

And remember - as Doctor Who puts it:

People assume that time is a strict progression of cause to effect, but *actually* from a non-linear, non-subjective viewpoint - it's more like a big ball of wibbly wobbly... time-y wimey... stuff. 

Enjoy your journey, enjoy your time :)

Technical Debt - from Owing to Owning

The term "Technical Debt" is used often in Scrum and otherwise Agile discussions. We know - or at least assume - it's a bad thing to have. Some best practices concerning definition of done for a story is to not leave any technical debt laying around. But what exactly is this technical debt all about? How does a team manage it? is it necessarily bad and should it always be avoided - at all costs?

Defining Technical Debt
Technical debt is any engineering activity that a Scrum or Agile team //should have done// but for whatever reason hadn't yet.

Some common examples might include:

• Smelly code that needs refactoring
• A broken automated build-server which while broken forces the team to do manual builds
• An automated acceptance test which is constantly and falsely red - not because there is a bug but because the test itself isn't properly testing the right outcome (a.k.a False Negative)
• A piece of code which doesn't have any tests at all

It's obvious from these examples that the team needs to do some work to get the debt "paid" - refactor the smelly code, fix the build-server, fix the test or write the missing test.

Work to do or Debt? What's the difference?
When I go to a store to buy groceries, I will take into account that whatever I'll buy will cost me money. - everything has a cost to it. Before I buy anything I don't owe anything, but once I collect my groceries and go to the cashier, I'm expected by mutual agreement and law to pay for what I've taken. But sometimes I find I forgot my wallet at home - and the shop owner might agree to give me the groceries and have me owe him money - that turns the cost into debt.

So back to software - when the Product Owner requests a story to be coded, during sprint planning the team will estimate and plan the total cost of the work - they will write down all the technical tasks that need to be done in order for the story to be considered "Done", and throughout the sprint will do everything in their power to complete all those tasks.
At this point the team has agreed - and committed - to do all the tasks. However sometimes (and sometimes more often than not) the team might find in the end of the sprint or even way after the sprint had ended that there are some "undone" tasks - things than still need to be done even though they should have been done before. This is where we view these tasks as not only "TODO Tasks" but as "Debt". The team owes the tasks to the PO or to themselves by power of their previous agreement and commitment.

Who do we Owe this to?
In Scrum the agreement on what tasks should be done for completing a given story are derived from the "Definition Of Done" (DoD). The DoD is an agreed upon list of criteria defining what it means for a story to be done - for instance it might include "there must be automated tests for all functionality", "all code should be refactored so there are no code smells left" and "All bugs should be fixed".
So if a team has "Technical Debt" - it's likely to be a violation of the DoD.
But the DoD itself is formulated through mutual agreement between the PO and the team, and will likely include both items that are important to the PO (like not having any bugs) and items that are important to the team itself (like having clean code or automated tests, since these will help the team move forward faster and make life easier for them down the road).
So any violation of the DoD is not a violation of a commitment to the PO, but in fact a violation of commitment to the team as well.
Owing technical debt is owing something to both the PO and to the team itself.

Don't get into debt in the first place? I wish...
One of the most common pieces of advice concerning technical debt is to not have it in the first place - I've heard consultants say that the PO should not accept stories for which all tasks have not been done yet, or that the team should not "claim points" for undone work. At some point the Scrum Guide included some pages dealing with "Undone Work" in it (I recall they have been removed in the latest edition, but don't take my word for it).

But in the end - there will always be "undone work" discovered somewhere - maybe at the end of the sprint the PO really needed that feature delivered to gain a business advantage and the team came to agreement with the PO that the code would not be fully refactored.
Maybe the team could get away with manual builds or manual tests for a sprint or two and investing in automation wasn't justifiable at the time, although now they realize they would be better off automating things.

Either way - technical debt (like financial debt) is usually brought to being through some agreement, done in good faith and for good reasoning at the time it was agreed upon.

Types of Technical Debt
I find most "Technical Debt" can be categorized in to one of three types:

1. Debt:
This is the work that we knew we needed to do but for some reason decided not to.

For instance, if I gave a dinner party at my home and decided not to wash the dishes - obviously I'll have to wash them the next morning.

In software, this might include:

• If a team had committed to writing automated tests but decided with the PO to not invest in them for the current sprint - they will probably need to catch up at some later time.

We call this Debt because the team had initially committed to do it but has later decided to postpone.

2. Discovered Deficit:
This is work we discover needs to be done only afterwards.

In my kitchen I might some day discover I have leaky plumbing. I didn't know of this ahead of time and had no way of foreseeing it, and will now have to take care of the situation before my next dinner party.

In software we often discover "Technical Debt" in hindsight - for example:

• While working on one part of the system we did refactor the code the best we could, but now - several sprints later - we have learned new coding patterns and realize that what was previously considered good code might be better if a new pattern was applied, and we want to refactor that code.
• We might realize we've been writing automated acceptance tests but in a form which is now very hard to maintain, and would be better off reorganizing those tests so they will be easier to maintain and evolve further.
• As the system runs throughout time in production (with real users) a database table might fill up beyond our initial expectations, and we realize that in order to meet our performance needs we should normalize it separating it into two separate tables (this is is fact a form of refactoring).

We call this Discovered Deficit because even though the team did everything they had committed to at the time, only with their newly gained knowledge, insights and perspectives can they now see and plan what they might have done better.

3. Technical Investment
Back in my kitchen - I wash the dishes, I have no plumbing problems, and the dinner parties are a great success - so I decide to buy a dish-washer to make my life easier.

In software, we might see opportunities to make the engineering work easier or better:

• We might want to install, learn and use a new testing framework, or better build-server 
• We might want to learn new techniques of working. 

These will presumably be good investments and although they will take some time (and maybe money) - they are assumed to pay off by improving quality or team velocity and so are good for business. Hence the term Technical Investment.


Is it bad?
Debt has a bad connotation - nobody likes being in debt, and if we are - we probably did something wrong. Once we realize that the technical debt was in fact agreed upon and taken into account - we can stop seeing it as a negative thing and start viewing it objectively - as mere "work to do".
Even calling this "undone work" smells bad - if it's "undone" it means somebody should have done it but didn't.
Looking at the categories mentioned above - we can see that much of the "Technical Debt" might in fact be the fruit of new insights or simply a worthwhile investment opportunity - not debt at all.

Seeking a more objective term - we might call these "Technical Tasks", or "Technical Stories", although these terms have their own baggage among Agile circles.

However we name it - this work differs from normal story tasks in that "normal" tasks (planned upon in the sprint planning meeting) are commissioned by the Product Owner ahead of time and are expected to create immediate value to the product - the PO realizes the tasks will need to take place in order to fulfill a story he/she wants done. The PO takes responsibility for the cost of these tasks as part of the responsibility for the product backlog.

The "Technical Stories" on the other hand are not going to be commissioned by the Product Owner anymore - the PO won't take responsibility for them, and won't prioritize them for upcoming sprints, since they don't drive immediate value to either user or customer. This means someone else needs to take responsibility for them - probably the team itself. They should be the ones wanting it to be done and pushing for it to be included in a sprint.

Responsibility and Ownership
Responsibility is a key factor to Agile software development - we want self organizing, responsible teams and individuals acting proactively and with respect to each other.
In Scrum the responsibilities are clearly divided - the Product Owner is responsible for the product while the Team is responsible for delivering it. This means the team is indirectly responsible for the quality of the code, and as a self-organizing and constantly improving team also responsible for it's own quality of work, it's pace, focus and capabilities.

Seeing that the team is responsible for their quality and performance, combined with the above-mentioned insight that the team in fact owes the "Debt" to themselves, turns the team into yet another stakeholder for the PO - a stakeholder that pushes to include "paying off technical debt" into each sprint.

So if the Product Owner //owns// the product, who //owns// the technical debt?
The team of course. They are owners of these "Technical Stories" just as the PO owns the "user stories". They own the list of "Technical Stories" (or "Technical Backlog") just like the PO owns the product backlog.

This puts a whole new perspective on what technical debt is - almost like the saying goes "it's not a bug - it's a feature", but in this case "it's not technical debt - it's a technical story".

From "Team" to "Code Owner"
Ownership of the product by a single responsible PO is what drives the Scrum process into creating working software that creates value to the business. Strong product ownership is based on drive, vision, a clear understanding of the product and business needs and the authority and capability to make decisions concerning prioritization and return on investment (ROI) of each feature or story.

Likewise, strong ownership of the "Technical Stories" should be driven by the same traits - a team should have firm belief in the need and in the gained value of completing each "Technical Story", should be capable of decision-making concerning which "Technical Stories" to do and when they should be done, and should be empowered to make those decisions - by mutual agreement and respect from the PO.

In this perspective the team is not only responsible for delivering working high quality software, but becomes the "Code Owner", equivalent to the "Product Owner" in the essence of ownership, while differing from the PO in the subject matter that is owned.

Viewing "Technical Stories" from this perspective enables a much more responsible, strong, proactive and positive approach to dealing with the tasks in hand than with the negative perspective of "Technical Debt". And this is the main motivator - not dealing with "why it happened", but rather with "how can we make things better".


Coming next.. "It's ours! Now what do we do with it?"
Once the team realizes that technical debt is not a bad thing but in fact something they own and are responsible for - they need tools and techniques for managing it.

I'll be covering that in an upcoming post. It's next in my backlog...