Predictive Power of Customer Metrics

[tweetmeme source="pricingright"]The usefulness of any customer metric depends on how actionable and how good a predictor of business success it is. Let us define here that business success refers to Sales growth and profitability.

  1. Of all the  metrics out there, is there one that serves as a good predictor of sales growth and profitability?
  2. Can there be really a single metric?
  3. What do you, as a small business owner, an entrepreneur or a decision maker for large enterprise need to know about the single metric trap?
  4. What other factors you should be aware of?

Read on.

Let us start with most common customer metrics, including but not limited to

  1. ACSI – Average Customer Satisfaction Index
  2. Top-2 Box (on a 5 point scale) Customer Satisfaction Score
  3. Number of recommendations – WoM, number of customers who actively recommend your product (service)
  4. Proportion of your customers complaining
  5. The Net Promoter Score

Supporters of some of the metrics claim theirs is the only metric any business need to track. In the data cited we will find a high positive correlation between these metrics and the two measures of business success. You do not need an advanced degree in statistics to question, “Does this correlation mean causation?”. But it does get a little tricky to sift through the data and flaws in analysis of the case for a single metric that predicts business success.

The biggest flaw that can occur in any argument that a single variable alone has predictive power is Omitted Variable Bias. Is there a lurking variable that was omitted in the model that drove both the metric and business success? This is not to say every argument that extends one predictor has Omitted Variable Bias but to raise the possibility that there may exist another variable that may explain the changes in your dependent variable.

Let me use an example to explain what it is before using it explain single metric trap.

This comes from Greg Mankiw. Suppose studies found a high correlation between test score of children and the number of bathrooms in their homes. Is this causation? Is this the single metric that determines success in tests? No. As Mankiw explains, the Omitted Variable here is the IQ of parents. It is possible that parents with high IQ earn high income and hence have large houses with more bathrooms. Their children may have high IQ because of the good genes passed on by their parents.

In the case of customer metrics, what could be the Omitted Variables? Some could be nature of products, your marketing strategy, channel strategy, nature of competition, etc. The question worth asking is,  Is the metric at hand with high correlation same as the number of bathrooms at homes? Let us take the third metric above, Number of Recommendations, as an example just for illustrative purposes. Is it possible that the nature of customers you are targeting have a high propensity to recommend? If you did not consider this possibility then you will incorrectly align all your resources and actions towards improving number of recommendations without any impact on business goals.

That would result in house full of bathrooms but still poor test scores.

I am not recommending that you give up on all metrics but  urge you to understand Omitted Variable Bias and consider the perils of tracking just one variable.

  1. What are all the different factors that are relevant to the business you are in and to your customers?
  2. How do these factors influence the single customer metric and your business success?
  3. After accounting for all these other variables, what percentage of changes in sales growth and profitability can be explained by the changes in that single customer metric you track?

In evidence based management any metric must be questioned for its predictive power and the methods by which the results are arrived at. Simplicity of a metric alone must not be the criteria.

Write to me, I will be happy to break this down more.


For a very readable and clear discussion of Omitted Variable Bias see also this post.

Analytics In Fantasy Football

Does the first week performance of your quarterback predict the performance for the rest of the season? This is a question posed and answered by Nando Di Fino in a column in The Wall Street Journal. I do not own a Fantasy Football league but I am excited to see analytics and econometrics being applied to player picks and trades. This is a very well written article and has some data analysis behind it. But I question the broad implications made by the article.

  1. Are the metrics used by CBSSports the right one to measure player   calibre and predict their performance?
  2. Is there omitted variable bias here? Could there be another  underlying factor that defines the overall season performance?
  3. I know Nando did not mean  causation but i think the   statement “Will a poor performance by a starting wide receiver
    foreshadow a season-long letdown? ” could be interpreted as such. If the  first week performance is indeed predictive of overall performance,  then the respective R-square values are very low (square of correlation coefficients) to fully explain the full season  performance.

This reminds me of a quote attributed Einstein, “Everything countable does not count and everything that counts is not countable”.