Making up things and supporting with faulty analysis

Update 7/11/202: I took a harsher stance against Mr. Jamison’s article. As I communicate with him over email and see his willingness to share data and refine his model I see my comments as little harsh. Instead of updating them I will leave them for what they are so you can also judge my writing.

How do VCs decide to pass on an startup? If you were to read a TechCrunch article you will find a quantitative model supported by statistical analysis:

Likelihood of Receiving Term Sheet = -0.355  +
0.349 (Team) +
0.334 (Market) +
0.222 (Traction) +
0.029 (Product)

A nice linear regression model with an R2 value of 0.5 that states Likelihood of getting Term Sheet as a function of four attributes. This article and the regression model comes from a partner in a VC firm, Mr. Jay Jamison.

Sounds plausible?  Fits your notion that VCs invest in teams and not product? Is the fact that this is a regression analysis done by a VC partner enough to convince you to suspend your disbelief and accept this predictive model at face value? Or are you going to walk up to the stage and tell the magician that you are not satisfied with his lift shuffle and you are going to do it yourself?

Let us do the latter and while we are up on the stage let us ask the magician to roll up the sleeves as well.

How did he build the model? Mr. Jamison, the author, said he rated each pitch on the five dimensions on a scale of 1 to 5. He explains more on how he defined the rating in his blog. Let us assume that it is interval scale to run Multiple Linear Regression (OLS – Ordinary Least Squares).

Now, what are the problems with this predictive model?

  1. How reliable is the data? Mr.Jamison collected 200 startup pitches available to him (not random sampling mind you) and ex post gave the rating. That is, these are NOT the ratings his firm gave on these dimensions at the time of the pitch but done by Mr.Jamison now just for the purpose of this analysis.
    That is a biased sample with flawed measurement. You can stop right here and call him out. The rest of the article and his claims based on the regression analysis are point less.
  2. How good is  the model? . A multiple regression model is measured by two metrics. One,  R2  which is the strength of the relation between the explanatory variables and the dependent variable and two a measure of whether each variable’s relation is statistically significant (p-value < 0.05)
    This model has an R2 value of 0.5.  This means 50% of the changes in Liklihood  (the Right Hand Side variable) can be explained by changes in these four variables. But is each explanatory variable’s relation statistically significant? Mr. Jamison does not provide us t-stat (or p-value) data for us. This is likely because he simply ran the regression with all the variables and reported just the R2 .
    If one were to use the simplistic Excel’s DataAnalysis tool to run multiple-regression that is what one will get. In essence, we do not know how many of the three variables really have any effect on the Likelihood of Receiving Term Sheet.
    The right way to do the regression is to enter variables one at a time and see if its relation is statistically significant and if the R2 value changes with the addition of variable to the model. It is possible only one of the variable is relevant and its  Rcould be much lower than 50%.
    So all the explanations on importance of Team, Market, Traction that Mr.Jamison provides are irrelevant because they are  based on faulty analysis.
  3. About the use of term Likelihood: It is misleading as I first thought he was really measuring Likelihood using Logistic regression. It is OLS where he models Likelihood on a 5 point scale. That rating is quite meaningless: it is simply a binary variable, whether he extended term sheet or not. In which case he should be running Logistic Regression which measures the probability that a startup will get term sheet given the values of four explanatory variables.

Even if the model did not have any of these errors, there are still lurking variables. Regression is not causation despite the equation form. It is still correlation and there are many lurking variables including who introduced the startup for the pitch and whether the VCs identify themselves with the startup founders.

What this really means is VCs don’t have any real model for evaluating startups.  Consider this – if we took this raw data, stripped out the Likelihood variable and asked VCs (in general) to rate the likelihood, how different are these going to be from VC to VC and how different will these ratings be from one done based on coin-toss?

It would have been interesting if VCs had a scoring system for these four attributes and other dimensions,  as a team rated the startups right after the pitch and agreed to extend term sheet to only those that reached certain threshold.

But what we have here  is faulty data and analysis used to color gut calls as quantitative.

Are you going to willingly suspend your disbelief? Or …

Leave it to Fast Company experts to find number one predictor of success

Fast Company has an FC Expert Blog. I do not know who these experts or what their qualifications are. They really are experts in declaring broad predictions, especially from reading few lines of some old academic paper. One of the experts write in their blog (the Fast Company says it is not responsible for their wisdom),

Grit: The Top Predictor of Success

Why do some companies consistently outperform their competition? Why do some people become champions while others fall short? What skills do you need to improve to reach your highest potential?

How ironic that a back-to-basics approach carries the day: It turns out that good old-fashioned grit is the number one indicator of high performance.

The experts, it turns out, did not read the details of the paper they quote. Nor do they seem to understand how predictability is measured in statistical terms and what it means. Needless to say they neglect to speak about omitted variable bias and other experimental errors.

What the paper says is grit, a trait defined by the authors, has an incremental R2 of 4%. That is when you add measure of Grit to whatever linear regression model they were building, the predictability of the model increased by 4%.

4%, just 4% increase after all other variables.

To go from here to “The Top Predictor of Success” is ludicrous.

Not just that, even the authors of the paper list severe limitations. The very definition of Grit is amorphous, it is highly correlated with the Big Five traits (classified in Psychology literature) and in their studies the authors measured it based on self-reporting by test participants.

From a study with such severe limitations (I am surprised it was even published), we get sage advice from Fast Company experts,

It doesn’t matter if you’re rich or poor, come from a good neighborhood, have a fancy-pants degree, or are good looking. We all have nearly limitless potential, and the opportunity to seize it is waiting for you.

Let old-school grit and determination serve as the catalyst to achieving your own personal greatness.  You don’t need another tech gadget; just the same killer app that has been foundation of success since the beginning of civilization.

The expert has filtered out gaping holes in the original study, ignored effect of lurking variables,  generalized a self-reported measurement of students to the entire population and urges us to show grit.

I grit my teeth!

Under every Complex Formula is …

Look at some of the quotes from the news media on companies with a data driven approach to management and business decisions

  1. Applying a complex equation to a basic human-resource problem is pure Google, a company that made using heavy data to drive decisions one of its “Ten Golden Rules” outlined in 2005. (WSJ on Google’s HR plans)
  2. Like Google, Facebook calculated the relevancy and authority of information before deciding to display it to me. The News Feed was shockingly complex — calculating and ranking more than a trillion items per day — and the results were very satisfying.  (WSJ on Facebook Newsfeed)
  3. Mr. Donahoe installed an entirely new system to determine which items appear first in a search. It uses a complicated formula that takes into account price and how well an item’s seller ranks in customer satisfaction. (WSJ on eBay)
  4. What could be more baffling than a capitalist corporation that gives away its best services, doesn’t set the prices for the ads that support it, and turns away customers because their ads don’t measure up to its complex formulas? (Wired on Google)
  5. CineMatch, on the other hand, is all math. It matches your viewing and rating history with people who have similar histories. It uses those similar profiles to predict which movies you are likely to enjoy. That’s what these recommendations really are – predictions of which movies you will like. (Netflix movie recommendation)

I am willing to bet that underneath the complexity is a multiple regression model, built with multiple variable and constantly tuned to better future behavior from past actions. Every business collects or has the opportunity to collect significant customer data. Companies like Google and eBay strive to be accurate 99% of the time or more. But building a regression model even with a handful of variables can improve decision making over driving without a dashboard.

Are your decisions data driven? If not I can help you build a model.