Handling Unplanned Work In Scrum

Before diving into how to handle unplanned work in scrum – Let us lay out what is: Technical teams who are building or supporting application software usually work from a prioritized list of improvements, new features, and ideas to try out. (Scrum calls the list the Product Backlog.) They have a plan for completing the items on the list. Except in cases when the totality of the work is completely understood in advance, contemporary practice involves rolling-wave planning or multi-horizon planning in some form, whereby teams pull a subset of items from the top of the list and work out the details, rather than specifying everything in the plan in detail up front. That way, they avoid investing too much time and effort in work items that are planned farther out in the future, so they can take lessons learned along the way into consideration when the time comes to address those items.

Things don’t always proceed as planned. People discover a piece of work isn’t as simple as they’d thought. People make mistakes. People get sick and miss time at work. People leave and are replaced with new team members who don’t know the code base or the organization’s internal procedures. Customers discover problems with the application and report “bugs.” Other teams in the organization request services of a specialist team with no advance warning.

All that unplanned activity used to create stress for technical workers. They had a plan, and their managers held them to the plan quite seriously. They lived in constant fear that something unexpected would come up and spoil the plan.

Fortunately, contemporary lightweight software delivery methods based on time-boxed iterations (like Scrum) or continuous flow (like Kanban) include mechanisms to handle unplanned work in scrum gracefully. Whether it’s a change in priority, a redefinition of a need, a reported defect, an unexpected request from another team, or a temporary change in people’s availability, reality must be accommodated but stress is no longer necessary. As the Zen folks like to say, “Pain is inevitable; suffering is optional.”

But…

That being the case, why do so many people panic every time something unexpected happens, such as unplanned work in scrum? When pain occurs, why do they opt for suffering?

Let’s explore some possible causes:

• Mindset: New methods overlaid on old thinking
• Cognitive bias: Reluctance to discard decisions or plans we’ve made
• Process: Uncertainty about what to do when unexpected work arises
• Band-aids: Working around the effects of the problem instead of improving the process

Mindset

“Stress is a result of disparity between expectations and reality.”

In the Old Days, the expectation was that building software was just like building anything else. You figure out what materials and tools you need and set up an assembly line. New software appears at the end of the line like clockwork. If that doesn’t happen, it means someone didn’t do their job, so you punish them.

Of course, that’s not how software is created, and it never was. Fortunately, we know that now and we can set our expectations to align with reality. Only…

…a lot of people don’t.

Don’t align their expectations with reality, that is.

Lightweight methods are designed to support an approach whereby we define a goal — a capability our customers or our organization needs in order to meet business (or other) objectives — and a direction for product design that seems likely to achieve that goal. Then we build incrementally and use tight feedback loops at multiple levels to steer the product toward the goal. We want to discover any problems early enough that we still have time and budget remaining to take effective corrective action.

The reality in that scenario is that things are going to change. Substantially. Frequently.

And that is what should happen. If people aren’t paying attention and actively steering the work, the end result won’t be good.

When people just ride along as if “the plan” is going to solve all the problems for them, but there’s no comprehensive plan because we’re using lightweight methods, then there will be misalignment between expectations and reality. There will be stress.

Cognitive Bias

We did our short-term planning last week and we came up with an idea of what work we would be able to complete in the next iteration or cadence. As we progressed with the work, we realized that things would not play out in just the way we predicted during planning. So we gracefully shift gears and move forward with the highest-value work that is actually possible to deliver, right?

Wrong. We bear down and set out to prove our initial plan was correct. We continue to blame ourselves when reality does as it pleases. People who have been using lightweight methods for years still talk about improving their estimation skills as a remedy for “failing” to force reality to conform with their predictions.

The idea of cognitive bias has burst forth from the social sciences to become a popular “thing” among lay people, like me. Reading through the ever-growing list of named cognitive biases, I’ve found three that seem to describe this behavior: confirmation bias, irrational escalation, and sunk cost fallacy. All those have something to do with our reluctance to let go of what we thought was right, even after we learn it was wrong.

Whatever you choose to call it, there’s a tendency for us humans to want our ideas and plans and predictions to be proven right. It’s counterintuitive for us to change our plans. Even if we understand on an intellectual level that a plan is no more than a best guess at a point in time, there’s something deep down inside us that is loathe to let go of what we thought was right as of yesterday or last week.

Process

“Even when people figure out change is normal and desirable, they often continue to feel stress whenever something unexpected comes up.”

If their mindset is okay, then what’s causing the stress?

It could be that they aren’t quite sure what to do. They aren’t clear about the process. Yet, most software teams are using one of two popular methods, or a combination: Scrum and Kanban. Both these methods include mechanisms to handle unplanned work in scrum gracefully.

People still feel stress when this happens. My observation is that most of the time it’s because of mindset: People expect everything to go according to plan, and they feel as if something is “wrong” when that doesn’t happen. They’ve signed up for contemporary methods, but they don’t quite trust those methods to do what they’re designed to do.

But I also see plenty of cases where people just don’t understand what to do. They most common cause in my experience is that they assume whatever they’ve planned for the next week or two absolutely must come true, no matter what.

But that isn’t how these methods work. Our short-term plans give us something to work toward, based on our point-in-time understanding of the universe. With every step we take, that understanding is subject to change.

When we’re using a time-boxed iterative process like Scrum and something unexpected comes up, the Product Owner prioritizes it and decides whether there’s a need to shift work around within the Sprint. Assuming the team is doing the right thing and not starting all the stories on the first day, there will be some stories that haven’t been started yet right up until near the end of the sprint. No effort has been invested in those stories yet, so there is no cost in dropping them from the sprint. If the Product Owner decides the Unexpected Thing is more important than one or more of those stories, then he/she drops it in and pulls the unstarted stories out. Simple. Painless.

Oh, but that’s not painless! you insist. We have a commitment to complete all the stories we signed up for. Here’s the deal: You have a commitment to do your best work to deliver the maximum value to your customers that is feasible to deliver. If something unexpected comes up and it’s of higher value than the work you had planned, then you meet your commitment by doing the highest-value work that’s in front of you.

With a continuous flow process like Kanban, the procedure is even simpler because you don’t have the overhead of managing sprints. You work on whatever is deemed most important today. You don’t have the emotional load of feeling as if you’ve “failed” to deliver something you “committed to” before you knew what Reality was about to dump on you.

Band-aids for unplanned work in scrum

From time to time, practitioners publish articles and blog posts that present some sort of complicated model of different kinds of “defects” and how to deal with them. What if the defect is detected within the sprint? What if it’s detected outside the sprint? When do we define a new task on an existing story, or define a new story? What happens to the work we had been planning to do, when we must turn our attention to an emergent problem? What happens when the actual time to complete a piece of work differs from our estimate? Lions and tigers and bears, oh my!

The suggestions you’ll usually read are along the lines of deciding whether the new work will add significant time to the story in progress, and possibly adding a new task to the story to handle it; possibly splitting the story and deferring part of the work to another time. That’s a workaround, not a solution.

So, a defect is found outside the sprint. How did that happen? What can we do differently going forward to minimize the occurrences of that situation? Programmers don’t usually write 2 + 2 = 5. “Defects” or “bugs” are almost always a result of miscommunication. Examine the communication model of the team.

What if a defect is found inside the sprint. What does that even mean? How can you have a “defect” before you’ve finished the story? The story is still in progress. You may have tagged it as “dev done, ready for test” or some such nonsense, but if the Definition of Done hasn’t been satisfied, then the story isn’t done. That’s a whole team responsibility. There’s no such thing as “dev done.”

The fact people are characterizing intra-sprint issues in terms of “defects” suggests something about team structure and team dynamics. An implication is there are hard hand-offs within the team and within the sprint. Another implication is there isn’t a clear Definition of Done, and maybe no Definition of Ready, either.

“There’s something in the process that’s causing churn. Find it; don’t just cover it up with a new task.”

The elephant in the room is this: Where are the defects coming from in the first place? What can we do differently to avoid creating them? Adding a task is all well and good, but how does that prevent the same pattern from occurring again?

Unexpected or unplanned work in scrum isn’t always due to “defects.” If you’ve chosen an “agile” approach for the work, it’s because “agile” is a fit for the situation. That means the solution has to emerge or evolve as more and more is learned about the problem through feedback loops. It’s only natural that things will change. Priorities can shift and stakeholders can discover new needs they hadn’t considered before, and determine that some needs they had believed were important aren’t necessary after all. That’s all good. It means people are paying attention to interim results and adjusting course toward the target business capability.

Organizational structure can lead to unexpected work items popping up, too. Some teams receive requests for services from other teams; for instance, an infrastructure support team, a database management team, or a team that supports a specialized third-party product such as a business rules engine or an ETL platform. Often, the needs of dozens or hundreds of in-flight projects in a large organization aren’t coordinated so closely that these specialized teams know in advance what requests are coming their way. If they work in a reactive mode, they’re likely to feel stressed out a good deal of the time. If they look for ways to smooth out the process, they’ll probably appreciate the results.

Conclusion

Every time something happens that feels uncomfortable (like unplanned work in scrum), it’s an improvement opportunity. Rejoice!