Recently, I analyzed past 10 years worth of 10-K filings of S&P 500 companies. A very interesting finding from this analysis is a high positive correlation (0.673) between number of times the word “customer” occurs in the 10-K and their profit growth. Every 20% increase in the word “customer” was associated with a 3.75% increase in profit. This finding adds to mounting evidence that companies that are customer focused stand to reap the benefits while those that are not are cast aside.
No wait. If you have not guessed already, I made all this up. But let us pretend otherwise and dissect this for analysis errors.
Correlation does not mean causation: I imply causation with my statement about 20% increase. Yes there may be correlation but it means nothing. It could be due to any number of reasons (lurking variables and Omitted Variable Bias. You can find many other correlations if you looked for it.
Meaningless metric: I nudge you to think that being customer focused is manifested in the word count.
Cross-Sectional Analysis: This means I looked across companies and found those with low word-count are associated with low stock growth and high word-count are associated with high profit growth. This ignores all industry specific and firm specific factors.
Implying Longitudinal Analysis: Longitudinal analysis is following a firm’s performance over years and see if the correlation holds true for a single firm over the years as they increased or decreased the word-count. However by stating, “10 years” I imply as if I just did this analysis.
Surviorship Bias: The imaginary data set consists of only those companies that are publicly traded and survived for 10 years and I only looked at S&P 500 companies, which are part of S&P 500 because of specific selection criteria . Half of the companies that were in S&P 500 10 years ago are not in it any more. So the sample set is even smaller. What about all other companies that are either private, did not make it or got dropped from S&P 500?
You probably did not worry about all these errors because the initial claim is so ridiculous that no further dissection was necessary. But not all claims on customer metrics are this obviously ridiculous.
There lies the danger!
These usually come neatly packaged and branded, come from some one with “authority”, stated with backing of data and vouched for by their marquee customers. They become extremely popular – accepted as gospel by other marketing Gurus and blogs. Authority and Acceptance become stand in for truth (Kenneth Galbraith).
But these claims are susceptible to exactly the same errors I stated above. I urge you to look beyond the surface and fanfare and look at the biases before you embrace the next big metric.