traffic lights are a great idea Decision support tools use colour coded icons to give you a quick snapshot of the “trends” in your measures, so at a glance you can decide whether to respond or not. Figure 1 is an example of a typical scorecard that uses traffic lighting.
Figure 1: excerpt from a scorecard with typical traffic lights
The traffic light icons are incredibly fast to interpret, much faster than reading the numbers and working out whether a negative % variance is showing a change in a good direction or a bad direction. This gives decision makers the ability to quickly focus on the areas of performance that most need their attention, and not waste precious decision making time on those areas of performance that are tracking well.
In theory, traffic lights are a simple and valuable technique to use in performance reporting.
but they are often very misleading
In practice, there is more to traffic lighting than meets the eye. One assumption that lurks unquestioned and unchallenged beneath the surface is this: two points of data are enough to determine if change has occurred.
The colour of traffic lights are almost always assigned based on the well used statistic called “% variance”. You'll see this statistic in almost every financial report and in the vast majority of non-financial performance reports too. It is usually the calculation of the difference between current actual performance and targeted or budgeted or last month's performance, expressed as a percentage of the target or budget or last month's value. So, if current actual performance is better than targeted or budgeted or last month's performance, the traffic light is set to green. If current actual performance is worse than targeted or budgeted or last month's performance, the traffic light is set to red. Sometimes a margin of tolerance is set, so that the difference between current actual performance and targeted or budgeted or last month's performance has to be greater than, say 5%, before the traffic light is set to red or green. In the case where the difference is there, but just not as large as 5% (or whatever the margin of tolerance is set to), the traffic light is set to amber. Of course, there are some variations to this approach, but the horrifying majority of these approaches are still based on the same assumption: two points of data are enough to determine if change has occurred.
Who decided that two points of data are enough to determine if change has occurred, and how did they decide this? It is a dangerous assumption, and it doesn't hold up in the face of either science or real life experience.
As far as science goes, the field of mathematical statistics can really be said to be the science of variability. It's all about trying to measure things that are inherently variable and drawing conclusions from collections of data by measuring this variation. In statistical terms, this variation is a measure of the amount of variability in a collection of data, and not the variability between two points of data. By understanding this variation, statisticians and other scientists have been able to conclude such things as which shampoos are safe for humans to use, that smoking increases your risk of lung cancer and how to increase success playing the stock market.
As far as real life experience goes, everything that we don't have 100% control over is inherently variable. Just think about how long it takes you to get to work each day. Or the exact amount of coffee you drink each week. Or the number of customer orders your company receives each month. Go ahead and list all the things in your life that vary as time goes by (that is, if you want to waste an incredibly large amount of time, because the list is endless).
Look at the historic values of your performance measures, and notice how they bounce around, up and down and down and up, as time has gone by. You don't honestly expect that they will follow a steady and unwavering linear path from where they are to where you want them to be, and then plateau to perfectly sit at the target value from that point forward, do you? Do you really have that much control over all areas of performance? Not likely. You will always be able to measure a non-zero % variance for your performance measures, because they will always vary to some extent.
So the majority of traffic lights are just telling you what you already know: performance varies.
they should tell you when performance is really changing
The value of traffic lights is not under question here. It really is useful to have quick and easy way to interpret signals that let you know when you need to respond, and when to leave well enough alone. And there's the magic word: signals. Signals of what, exactly?
Signals of change. Signals of whether performance, despite it's inherent variability, has gotten better, gotten worse or is staying the same. Signals of whether your efforts to manage performance have worked or failed. Signals of whether you should keep doing what you are doing, or do something different.
Is a measure of “% variance” a real signal? No, because it doesn't tell you how much of the variance is inherent and normal, and how much of the variance is due to some change and thus special. Signals come from the special variation, not in the normal variation. If you react to “% variance” values, you run a very high risk of missing true signals that are really there, and wasting resources when true signals are not really there.
A better assumption to drive your setting of traffic lights is: you have to discern abnormal, special variation from the normal, inherent variation, to see the true signals.
Like the following example, it becomes really evident why “% variance” is not a reliable indicator of change in performance. Notice in Figure 2 how looking at the variability over time of the performance measures from Figure 1, you draw completely different conclusions. The charts used here are statistical process control charts, designed by Walter Shewhart in the 1920's specifically to help discern normal variation from abnormal variation. The idea is to look at changes in the pattern of variability compared with the straight mean line and upper and lower limits of variation.
One of the interpretation rules of these statistical process control charts is if you see 7 or more points in a row that are not randomly varying around the mean line and inside the limits of variation, then something is changing. Interpreting these charts, you can see that the traffic lights in Figure 1 are giving entirely the wrong signals!
Figure 2: excerpt from a scorecard with valid traffic lights
All this means that there are better ways to set your traffic lights, so that they are giving you true signals about when performance is really changing.
designing traffic lights that work
Measuring and monitoring the variation in your performance measures over a decent window of time is the key to identifying the true signals. Here are some tips for getting beyond point-to-point comparisons of your data and getting closer to tuning into the true signals of change.
about the author
Stacey Barr is a specialist in performance measurement, helping people to move their business or organisation's performance from where it is, to where they want it to be.  Content date: Friday, July 11, 2008
Author: Stacey Barr (info@staceybarr.com ) Company: Stacey Barr (http://www.staceybarr.com) « back |
|||||||||||||||||||||||||||||||||
2004 © artMania internet solutions | All rights reserved