I turned a fellow Dashboard Spy reader onto a great primer on setting business metrics and I’d thought I’d share it with you. Download this word document – it’s Chapter 3 and Chapter 4 from the Greg Brue book “Six Sigma for Managers”.

Here’s an excerpt that I always take to heart. There can be significant problems with business metrics if you are not careful:

Problems with Metrics

It’s easy to get caught up in measuring things, so that you focus so much on quantifying defects that you forget about also quantifying the effects. Here’s an example. If you’re setting metrics for your secretaries, you might include a measurement of typos in terms of 1000 characters (opportunities). So you determine that George averages 5 typos, Sarah averages 7, and Pat averages 8. Well, obviously George is the most accurate, right? Yes, if you quantify defects only. But what about the effects on your customers? George generally has more problems with names, while Sarah and Pat check names carefully. Since customers are usually more bothered by mistakes with their names than mistakes with other words, George would suffer by comparison with Sarah and Pat. That’s one of the problems with being too focused on counting defects alone.

Along these lines, we should caution against focusing on averages. The usual way to represent a series of figures is by finding the average. But consider the potential complication. Here’s an example. Your goal is on-time deliveries. For your three delivery drivers this month, you calculate averages of 15.3 minutes late, 24.7 minutes late, and 6.3 minutes late. So, you conclude, Driver 3 has the best average. That’s true, but the averages don’t show everything. They don’t show that Driver 3 is often as late as Drivers 1 and 2, but occasionally arrives 20 to 30 minutes early. That helps compensate for being late-but it inconveniences the customer when dock workers have to cut their lunch short to unload the truck. The averages also don’t show that Driver 1 has several times been late by 45 minutes, while Driver 2 has been late by 20 minutes at most. Six Sigma allows you to measure variation in a process, to calculate standard deviations from the mean, so you have a more accurate picture of the process.

A final point to make here is that metrics should use units that everybody understands. If, for example, we want to establish metrics for incomplete shipments, what constitutes “incomplete”? Does it matter how many items are missing? Do you account for the relative importance of the items to the customer? If so, how? The problem of an incomplete shipment is worse if the customer refuses delivery, but how do you measure that effect? How do you establish a metric that doesn’t require any of the employees to make judgments when they track the data?

How do you establish appropriate, accurate metrics? You tap the experience of the employees who are closest to the processes. You hold meetings to discuss your attempts at establishing metrics and you encourage everyone to find fault with them. Then, finally, as you use your metrics, encourage one and all to report any questions or problems with them.