Leave it to Fast Company experts to find number one predictor of success

Fast Company has an FC Expert Blog. I do not know who these experts or what their qualifications are. They really are experts in declaring broad predictions, especially from reading few lines of some old academic paper. One of the experts write in their blog (the Fast Company says it is not responsible for their wisdom),

Grit: The Top Predictor of Success

Why do some companies consistently outperform their competition? Why do some people become champions while others fall short? What skills do you need to improve to reach your highest potential?

How ironic that a back-to-basics approach carries the day: It turns out that good old-fashioned grit is the number one indicator of high performance.

The experts, it turns out, did not read the details of the paper they quote. Nor do they seem to understand how predictability is measured in statistical terms and what it means. Needless to say they neglect to speak about omitted variable bias and other experimental errors.

What the paper says is grit, a trait defined by the authors, has an incremental R² of 4%. That is when you add measure of Grit to whatever linear regression model they were building, the predictability of the model increased by 4%.

4%, just 4% increase after all other variables.

To go from here to “The Top Predictor of Success” is ludicrous.

Not just that, even the authors of the paper list severe limitations. The very definition of Grit is amorphous, it is highly correlated with the Big Five traits (classified in Psychology literature) and in their studies the authors measured it based on self-reporting by test participants.

From a study with such severe limitations (I am surprised it was even published), we get sage advice from Fast Company experts,

It doesn’t matter if you’re rich or poor, come from a good neighborhood, have a fancy-pants degree, or are good looking. We all have nearly limitless potential, and the opportunity to seize it is waiting for you.

Let old-school grit and determination serve as the catalyst to achieving your own personal greatness.  You don’t need another tech gadget; just the same killer app that has been foundation of success since the beginning of civilization.

The expert has filtered out gaping holes in the original study, ignored effect of lurking variables,  generalized a self-reported measurement of students to the entire population and urges us to show grit.

I grit my teeth!

3 Factors that Drive Customer Satisfaction Rating

When it comes to customer satisfaction rating, more of everything isn’t the answer. From regression analysis of years worth of customer satisfaction rating and from related works done by others, we find that customer satisfaction is driven by 3 basic factors (from stated rating studies):

1. Buying experience: How easy it is to evaluate choices and complete the buying process? Customers treat buying experience as part of the product experience. While rational thinking dictates that these costs are incurred once and should be treated as sunk by the customers, research(Journal of Management Information Systems Winter 2007-08) shows that these costs remain sticky and customers treat buying experience as part of the product experience.
2. Delivering what is promised: Does the product quality and its realized benefits match what was promised and most importantly what the customer expected it to be? This is not about delighting the customers are delivering more that what is promised. A customer who walks into WalMart has one set of expectation and the one who walks into Nordstrom has another. For the segment you are targeting, the product benefits must match your positioning and messaging.
3. Experience when things go wrong:  In the case when things go wrong, customers need support, how easy it is to get support and how they are taken care of. No customer believes things will never go wrong but the type of support they receive and how the problems are handled are what customers treat as relevant to their overall satisfaction rating. For example, a Corolla customer does not expect the dealership to send a loaner car and tow-truck for services, but a Lexus customer does.

Go head test this out today. Run a very simple survey of 4 questions to your customers, (use 1-10 scale)

1. Please rate your overall satisfaction rating with our products and services.
2. Please rate how satisfied you are with your buying experience (how easy it is to find what you need, evaluate options and complete the buying process)
3. Please rate how satisfied you are with our product quality (meeting your expectations, delivers what was promised)
4. Please rate your support experience (ease of getting help, timeliness, how you were treated)

Run a regression using (1) as dependent variable and the rest as independent variable and you will find out how relevant the 3 factors are to your own situation.

Caution: Regression analysis still only finds correlation. There are numerous lurking variables that were not fully studied. But research from other data sets make it more likely that these variables have causation relation to customer satisfaction.

Implying Causation – Predictive Analytics Slippery Slope

Imagine, if you will, a child eating broccoli for the very first time. While eating broccoli, let us say the child sneezes a few times in succession and then proudly declares, “I think I am allergic to broccoli”. As a parent or simply as a grown-up it is not difficult for you to see the fallacy in child’s case. One does need an advanced degree in econometrics or statistics to  reply back, “eat your broccoli – correlation does not imply causation”.  Consider the following real cases:

1. From The Times Economix Blog:

There’s a very strong positive correlation between income and test scores. (For the math geeks out there, the R² for each test average/income range chart is about 0.95.)

2. From The WSJ opinion column:

Study after study reveals that there are long-term career benefits to working as a teenager and that these benefits go well beyond the pay that these youths receive. A study by researchers at Stanford found that those who do not work as teenagers have lower long-term wages and employability even after 10 years.

3. From WSJ half-page Ads targeting parents

Students who read The Journal are 76% more likely to have a GPA of 36% or higher

4. From a research paper on subscription to library resources by universities

Working with Dr. Carol Tenopir of the University of Tennessee and consultant Judy Luther of Information Strategies, this single-case study demonstrates a $4.38 grant income for each $1.00 invested by the university in the library (ROI Value). The white paper External link University Investments in Information: What’s the Return? is posted on Library Connect. The results articulate the relationship between the value of research information and its impact on the funding of an institute.

5. From a research paper from the London School of Economics

In terms of percentage growth, a 7 point increase in word of mouth advocacy (net-promoter score)
correlated with a 1% increase in growth (1 point increase = .147% more growth). The measurement was done through telephone survey in 2005 and the revenue growth numbers are for 2003-2004.

Can you spot the fallacies in these claims?  Are these seemingly erudite and well researched claims any different from the claims of a smart child that wants to avoid broccoli? Why do we want to see correlation when none exist or take correlation for causation? Why do we suspend our critical thinking when the results are presented by big brands, big universities and packed with tonnes of data and graphs?

Of all these cases I listed above, the last one is the winner. Suppose in the chronology of events,  event-2 follows event-1 in time. It is pardonable and a ubiquitous mistake when someone says event-1 might have caused event-2. This is the garden variety correlation causation confusion. But this example I quote says, “event-2 caused event-1”.

I do not know a word for this!

Analytics In Fantasy Football

Does the first week performance of your quarterback predict the performance for the rest of the season? This is a question posed and answered by Nando Di Fino in a column in The Wall Street Journal. I do not own a Fantasy Football league but I am excited to see analytics and econometrics being applied to player picks and trades. This is a very well written article and has some data analysis behind it. But I question the broad implications made by the article.

1. Are the metrics used by CBSSports the right one to measure player   calibre and predict their performance?
2. Is there omitted variable bias here? Could there be another  underlying factor that defines the overall season performance?
3. I know Nando did not mean  causation but i think the   statement “Will a poor performance by a starting wide receiver
   foreshadow a season-long letdown? ” could be interpreted as such. If the  first week performance is indeed predictive of overall performance,  then the respective R-square values are very low (square of correlation coefficients) to fully explain the full season  performance.

This reminds me of a quote attributed Einstein, “Everything countable does not count and everything that counts is not countable”.