But is this chart really saying this? Remember, it's a rolling average - the value for May 1999 is the average of the last 12 months' results, from June 1998 to May 1999. So that downward trend that your EYE sees, is not describing what's happening in the period from May 1999 to March 2000, it's describing what has happened in general since June 1998!

Is this really the best description of how reliability is performing over time?

Have a look at the next graph, which is based on exactly the same reliability data, but shows the actual monthly totals instead of the rolling 12 month average of these values. The chart type used is called a statistical process control chart. The shaded area corresponds to the "limits of normal variation", which is a statistically calculated range around the performance values, within which you would almost always see the values fall. This shaded area is a measure of how much variation there is in performance, and thus gives us some idea of its predictability into the future.

This alternative way of representing performance data tells us some different things about reliability performance. Firstly, it doesn't show any kind of gradual decline at all! It does, however show us that in April 1999 there was a sudden drop in the overall level of reliability, and an associated narrowing of the natural variability in reliability. It doesn't suggest in any way that reliability is getting worse.

Well, two very different interpretations of exactly the same set of data, but two different methods of analysing that data. Why did this happen? Remember that rolling averages were designed to smooth out seasonal fluctuations to reveal the overall trend. Is that an appropriate analysis method for this set of data, given that there is no seasonal pattern? Is the overall trend the only thing we need to know in order to manage performance like this reliability result? No, on both counts.

If you use rolling averages (or other forms of cumulative analyses) for your performance measures, you miss out on essential information like:

• what kind of changes have occurred over time, including sudden shifts in results, changes in the amount of variability, as well as the gradual trends
• when exactly the change occurred (eg when it began, when it ended)
• how big the change actually was

And aren't these the basic things any decision maker needs to know before he or she can decide on a sensible course of action to identify and manage the causes? Reducing all our performance results to just overall, gradual trends means missing out on a lot of other important signals that we should be responding to.

Rather than using rolling averages, why not try some analysis methods that will better highlight to you all the important types of signals your performance data could show you? To do this, you need to adopt the following practices:

• use the actual values of your performance measure each time period (such as week, month or quarter), not a rolling average or cumulative value
• use a line chart format to plot your actual performance values, for as long a timeframe as you can (ideally 20 or more)
• look for patterns in your time series of actual values, more patterns than just gradual increases or decreases
• look for sudden shifts up or down in the overall level of your performance results, look for sudden or gradual changes in the width of variation over time, look for cyclical patterns, look for outliers (those points that are exceptions to the rule)
• try using statistical process control charts, which help you identify and highlight a range of typical patterns (in addition to the gradual trend) in your performance results

And if you are still not convinced, then try using both types of analysis on the same set of data, and just observe how you interpret each one, what signals are highlighted and what actions these signals suggest you take. Appropriate analysis and interpretation of our performance data is just too important to stay complacent with traditional but ineffective techniques.