Gaming the Numbers
I’d like to see if we can generate a little discussion around the idea of metrics, goals, and improvement.
Yesterday I was exploring with a client the notion of quarterly objectives. Specifically we were discussing the kinds of metrics that would make sense in an agile environment to properly incentivize the kinds of outcomes we are really going for.
Out of that conversation came the best quote of the week, maybe the best quote of the year… “If you don’t want people to game the numbers, don’t make the numbers a game”. I wish I could take credit for that one, but credit goes to the client ;-)
So… here is the question. What are some ways to measure team performance without creating an incentive to game the system? What kinds of things do you guys measure and track that allow you to baseline, track, and reward authentic improvement?
I think it would be neat of we could get an inventory of ideas in the comments to this post. I’ll add mine… you add yours… and let’s see what we come up with. If you guys hurry, maybe I’ll share your feedback with the folks at the Agile Coach Camp tomorrow!
Comments (14)
David Koontz
Interesting question – although I’ve not yet tried this, I wonder if a form of “net promoter score” would work.
http://en.wikipedia.org/wiki/Net_Promoter
I would see it working for large groups – say 100+ developers. Survey the members asking them the question:
“How likely is it that you would recommend our company (and your current team) to a friend or colleague looking for a job?”
To get even fancier I’d attach that question to the Release retrospective. Plot the trend over time. It is a measure of team happiness.
James O'Sullivan
For me the most important metric is customer satisfaction. This also can’t easily be gamed.
Would you guys see customer stat scores or netpromoter as a leading or lag indicators? What if the customer isn’t so agile friendly and dings the team through no fault of their own?
Bruno Bougie
I would agree customer satisfaction is probably the most important indicator. However, to me it’s not always clear who “the” customer is… is it the representative of the company who decides and signs the contract, or is it the employees of the company..or maybe the end user (customers of the customer)???
I’ve worked for a global IT service provider and the customer satisfaction interviews often turned out to be… well useless… when it was a company wide questionnaire the employees might not be too happy about the time it took their problems were solved..but the IT managers were happy because they could report to their HQ’s costs were decreasing… when there were one-to-one interviews to keyplayers, we found theinterview was often used as a political/commercial tool….
so I think defining “the” customer is essential.. it wouldnt surprise me if it turned out within your own team people have different views on who the customer is. Once you have that sorted out…you could decide what the metric of csat is… or better… have a session with the customer… I think most customers are willing and will be honoured to help you discovering what the best way of measuring their satisfaction is.
At my current company (insurance) we have “customer arena’s” …. customers are invited for a couple of hours to give feedback…
first phase: the customers sit in a circle facing eachother and the supplier (we) sit in a circle around them. There is a person leading the conversation and the customers tell there experiences… during this phase we are NOT allowed to talk…just listen. The person leading the convo has to be skilled in getting out the real issues that are important to the customers ..because there are several customers they feed each other and a lot of stuff comes out.
during the second phase.. we are allowed to ask the customer questions to get things clear… we are not allowed to defend or attack. We are also allowed to ask the customer other questions we would like to know their opinion about.
The third phase is without the customer..we decide what we are going to do with the feedback…depending on the subjects you could involve the customers again later in the process to see if your solutions match their expectations.
I think most important indicator is: would the customer recommend you to someone else…
Bruno, do you think there is any room for a quantitative, less subjective measure?
Marcin Floryan
Why do you want to measure performance would be my first question. Is team performance (if we can put any kind of number against it) really that relevant? I’m afraid that if you want to tie performance with rewards then you’re limiting what the team can do. They might not be able to game the system if you come up with something really sophisticated or simple and transparent but they will focus on that measurement.
In my opinion, the only thing that matters is whether you are delivering software that delights your customer.
And please, don’t consider rewards (extrinsic motivation) as a driver for improvements. Their effectiveness is very limited. On the other hand, please do reward your teams for good work – in a specific, timely fashion and always after the facts.
Marcin, you focused on the word performance. How about improvement? I get the idea of customer sat, but it feels subjective, seems to be a lagging indicator, and worse… may lead to lack of clarity around who is the customer. Thanks for your feedback, but I’m struggling to see a clear path to measuring customer satisfaction and how we would use that to get better. Also, I’m a Dan Pink fan and get the extrinsic motivator thing. Are you suggesting we shouldn’t tie any amount of compensation to any measure of performance?
Bruno Bougie
I guess whether there is room for quantitative measurements depends on the level or viewpoint you have. Before I continue I think I should say that I personally I have close to 0 (zero) experience with software projects.. so my examples might be a bit off..but I am sure you’ll get my point.
If you want to measure quality / innovations / productivity it would be good to think about the question whether those concepts are absolute. For instance, In software I could imagine that user convenience and security are conflicting. So from a security point of view you could argue things are getting better…however users get increasingly frustrated by all the passwords and authorisation steps in the process… If you would want to measure overall improvement, you would have to put a weight-factor to security and user-convenience to add it up.
I think those factors will always be subjective… but even if you would manage to come up with objective measures… the end result you calculate is “just” a weighted average..which, from a statistical point of view might be valid…but no actual person recognises him/herself in the figure… so the question is: what’s the value of that calculated number.
same goes for the net promotor score you mentioned earlier… you might have a positive promotor score…but the actual detractors might have much higher influence than the promotors… so what does the NPS tell you then?
anyeway, yes, i think when you get your scope narrow enough… you can measure improvement rather accurately … to assign a number to overall improvement..is hard (to say the least) …
another thing that I think is import is whether the improvement viewed from a developers point of view, or from the customer/rest of the world point of view. In my previous job i used to be responsible for a part of the (financial) admin. We used an Oracle db and had excellent database developers…. whatever we wanted… they found a way to make it work… after two years we basically created…a monster… the admin had gotten so complex only a few people had the total overview…if we had to change something it took days of meetings to get it clear what needed to be done… Now if we would have measured the innovation/improvement those db programmers did..I think they would have scored a very good figure…if you would have asked us (customers) we would have said we were happy with them…. but looking back now, I would say we made a major mistake and should have gone for simplicity from the start…you don’t have to build anything because it can be done.
Same goes for measuring improvement/innovation … you could make it very complex and theoretical…but in the end I guess it’s just a matter of asking a few key players (customer-users, customers-management, customer-executives, own org-operations, own org-management, own org-finance) how they feel and if they think you are on the right track and you are done.
Rob Bowley
I’ve regularly found cycle time to be a good measurement for these types of things. The metrics I find useful are average cycle time (which ideally goes down), variation via std. dev (which shows how predictable you are) and throughput (the more you’re getting done the better). All these three metrics need to be used together to be meaningful.
To ‘game’ cycle time you just need to split work into smaller more releasable pieces. Surely not a bad thing.
Steve Paro
First let me start by saying all metrics need to be team based. Given that, I think it could be interesting to measure impediment/issue cycle time. An improving team should over time get better and faster at resolving their issues. Even for those issues that fall outside their normal sphere of influence as they should learn creative ways to work around many of their so-called organizational constraints.
Also, how about measuring retrospective action items. How many opportunities for improvement were identified and how many acted upon? In this case, I’d be more interested in the number of improvement evolutions than the actual results of said improvements.
Evan Leonard
Why not let the team define their own metrics? Say once a quarter, have each team choose areas for improvement and a metric to measure by. Then they can include this metric in their retrospectives for three months. And then change the game again after three months.
Evan Leonard
Also, funny that the Nokia test hasn’t been mentioned here. What I’m currently thinking is that our teams will have to pass a baseline score on a “nokia-style” test before they are allowed to define their own metrics. I believe that you have to establish a common floor of behavior before the system will generate the type of creativity that we’re looking for.
Matt Block
I definitely agree that all measures need to be team based and you also need to be very clear about the point of the measurement. Are you creating a goal or are you just looking at an indicator? If you really want to create a goal, make sure that is something you really want to optimize, potentially at the cost of just about anything else.
That said, I do think there are some things you can look at. As others have said, I think customer satisfaction is a very good measure. There are some issues as have been discussed higher in the thread, but if you can work through those issues, this is one of the best measures of success. Even better than satisfaction may be delight and/or Net Promoter Score (is it enough to just satisfy your customer, or do you really want to delight them to the point that they will be out promoting you in their networks). Revenue/Profit are also great indicators, but can be even trickier to track than customer satisfaction especially when you are trying to value incremental improvements (was that increase in revenue really because of this new feature). Something you can look at that can generally be a bit more fine tuned than revenue is simply usage. Maybe you added a new feature and it’s hard to tell if it is driving new revenue, but you should be able to see if people are using that new feature. If that new feature provided enough value, people should be using it. Similarly if you are in installed software world, if your latest upgrade provided enough value, your customers should be upgrading.
As for leading indicators, Scott Downey (http://www.rapidscrum.com) has some great metrics like his re-defined focus factor (how much of the team effort is actually resulting in requested, accepted working software at the end of the sprint) which is a good measure of efficiency. You can also look at things like his measure of Found Work along with estimation accuracy as indicators of how much shared understanding the team has.
Generally speaking, I’m a big fan of measuring all sorts of things and looking at those numbers. I’m generally not a big fan of tying “goals” to those numbers, especially with monetary rewards. And I’m firmly against trying to look at those numbers on an individual level, they should definitely be at a team level. I was in the process of my own blog post about Agile Metrics when Mike posted this, so you can check out my thoughts there…http://www.developmentblock.com/2011/09/agile-metrics/.