2.022010

misleading metrics

Ever since the phrase “you manage what you measure” made its way into the dialogue of corporate America, metrics have played an increasingly important role in the management of businesses.  measurement cartoonNo one can argue the value of timely, detailed, and accurate measures of performance as assessments of effectiveness and guides for decision-making.  Yet, which metrics are best remains highly debatable.


Lately I’ve been troubled by two metrics.  I’d like to outline my concerns and then hear your feedback, as I’m sure not everyone will agree with them.

First, there’s ROI.  While return-on-investment is a legitimate and helpful metric for evaluating the efficiency of an investment in finance and economics, I’m concerned about it as a method for measuring the effectiveness of many business moves.

I see ROI as a tool for use in the capital budgeting process – leaders must make projections of future rates of return of different projects in order to select which projects to pursue.  It also has value when evaluating marketing outcomes at a high level, comparing, for example, two different product launches by calculating the revenue that each new product generated divided by its respective marketing expenses.

However, I question whether ROI is the best way to measure discrete tactical activitiesChristine Crandell, EVP and CMO of Egenera, explains my concern well in a piece published by CMO.com:

“The problem with ROI is that it forces a myopic view on short-term objectives while disregarding the fact that it is very rare for any single marketing activity to be directly responsible for new net revenue.  The focus on ROI fuels a never-ending debate about spending levels for brand-building activities while disregarding the fact that customers typically buy only after being repeatedly touched by a series of separate and discrete marketing activities.” (emphasis mine)

She points to the current debate about the ROI of social media to show how such a focus can be so misguided. She also suggests an alternative:

“…Replace ROI with benchmarks.  Benchmark marketing’s performance against itself quarter over quarter and year over year.  Benchmark against comparable competitors.   Benchmark the number of leads passed to sales that closed, the average revenue per lead, the percentage of the forecast generated by marketing, the percentage of customers actively engaged in advocacy programs, or the number of lead referrals from industry analysts.  And so on.” (emphasis mine)

This approach makes sense to me, as it focuses metrics on specific measures that can be compared apples-to-apples. And rather than simply offering a referendum on an outcome that happened in the past, it provides more direction for the planning of future strategies and tactics.

ROI also isn’t the right metric for measuring much of the work that brand-building practitioners like myself do. Even the most bottom-line oriented business person would be hard pressed to calculate the ROI of identifying the optimal competitive positioning or implementing a customer experience strategy to increase the consistency of value delivery.  Trying to attach a sales or profit result to changes that have such foundational and broad-sweeping effects would be futile at best, misleading at worst.

That’s not to say that measuring the impact of such initiatives is unimportant.  An evaluation of the value created by them is more fitting.  Last year, I wrote a series on brand value creation which outlines the different outcomes produced by a strong brand, including price premium, business process efficiency, and stakeholder engagement. It’s measurements like these that would be a more accurate and direct reflection of brand-building effectiveness.

The other metric that has me concerned is comp store sales.  Comp store sales, which indicates the growth or decline of sales of stores that have been open for a year or more compared to the same period the year before, is the primary means of reporting retail results.  It’s an important measurement of operational performance because it distinguishes between revenue growth that comes from new stores and growth from improved operations at existing outlets.

Problems arise though because a strong comp store sales result this period may reflect poor performance last period, not necessarily good performance in this one.  No doubt, many retailers’ comp stores sales are going to start looking good later this year as we start rolling over the depths of last year’s recession.

Furthermore, longer quarters and holidays falling in different quarters can have a significant impact on comp store sales.  And higher comp store sales can often be a result of price increases, not unit sales or traffic increases.  That’s why I cringe when I see people placing too much emphasis on monthly or quarterly comp store sales, as they did this past holiday season when repeated reports of increases were released.

Perhaps benchmarks make sense here as well.  The focus on comp store sales seems more appropriate if we use the metric to benchmark one company’s results against its primary competitors and against the industry average.  Instead of reporting comp stores sales as a percentage of growth or loss, perhaps it should be reported as a score relative to an index.

Also looking at trends vs. absolutes is more revealing.  If a company has been experiencing a steady increase in comp store sales over multiple quarters and years, it’s clear that operational improvements — not flukes or fads — are driving the results.  Wide variations from month to month or quarter to quarter, however, indicate a much less stable business.  So perhaps tracking deviations from a mean expected increase would be a meaningful metric.

I don’t mean to make measuring performance more complicated – it just seems the over-reliance on one number, whether it’s ROI or comp store sales, can be dangerous.  And in the end, the point of a metric is not the number itself.  Rather, a good metric is a tool by which future decisions are guided.

So to the adage “you manage what you measure,” I would suggest the following corollary be added: “you manage well what you measure well.

I’m eager to hear your thoughts on this post. Comments are open.

related posts:

, , , , , ,