Sometimes it helps to understand data representation and statistics than just eye candy design
The post is titled, “10 horrifying stats about display advertising”. In a attempt to tell stories or relate arcane data to something common the author goes on to make some likelihood comparisons
You are more likely to complete NAVY SEAL training than click a banner ad.
You are more likely to get a full house while playing poker than click on a banner ad.
You are more likely to get a full house while playing poker than click on a banner ad.
You are more likely to birth twins than click a banner ad.
You are more likely to get into MIT than click a banner ad.
You are more likely to survive a plane crash than click on a banner ad.
It is pointless and simply wrong to make such comparisons based on respective frequentist probabilities. Let us say there is a one tenth of one percent of people who see display ad click on it. Let us us say one percent of people who sign up for Navy SEAL complete the training.
That does not mean these two are comparable nor can you say that chances of ANYONE completing Navy SEAL training is far better than clicking on display ad. What is missing here are the hidden hypotheses we take for granted (the context).
Think of those who sign up for Navy SEAL. Consider their drive, motivation, physical fitness, mental strength, initial screenings they survived to get to training stage. You already have weeded out people like us. If indeed 1% of people who attempt SEAL training complete, it is the conditional probability
P(Complete SEAL training | Passed all screenings and have wherewithal to complete it)
This is not the probability of any random person you pick from street, which is highly likely close to 0.
It is indeed horrifying that they would compare such unrelated events and their conditional probabilities to make their case about display advertising.
Now to the title of this article. Determining whether or not this is relevant comparison of likelihoods is left as an exercise to the reader.
Most people attribute this quote to Einstein,
Not everything that counts can be counted, and not everything that can be counted counts
It is repeated with deference by most. There are however alternative theories about whether or not this quote should be attributed to Einstein or not. That is not the argument here.
The argument is how statements like
“not everything can be quantified”
“not everything can be measured”
“you have to look at this holistically”
“we have been in this business long, we know”
get thrown around to support decisions made only with lots of hand waving and posturing. Anyone pushing back to ask for data is told about this famous quote by Einstein – “surely you are not smarter than Einstein”, is the implied message I guess.
Let us treat that famous quote as given – there are indeed factors that are relevant to a decision that cannot be quantified. But ..
- How many business decisions depend on such non-countable factors? Consider the many product, marketing, strategy, pricing, customer acquisition etc. decisions you need to make in your job. Do all of them depend on factors that cannot be counted?
- Considering many factors that go into making a decision what is the weight of non-countable factors in any decision? You don’t say 100% of any decision depends on non-countable factors.
- How many times have you tried to push through decisions based entirely on non-countable factors (stories?) with not an iota of data?
- Finally, have you stopped to check whether what you label as non-countable is indeed so and is not so because of your lack of trying? Are you casting those fators as non-countable either because you are not aware of the methods to count them or because you are taking shortcuts? If you made an attempt you will find most things are countable (How to Measure Anything), be it segment size, willingness to pay or brand influence.
I cannot speak for what Einstein (or someone else) meant with that quote about non-countable things that count. But I am going to interpret that for our present day decision making.
For any decision there are factors that are inputs for it. Some of those factors can be measured with certainty, some with quantifiable uncertainty* and others with unquantifiable uncertainty*. You must cover first, have a model for second and put in place methods to learn more to reduce the third.
Without this understanding if you try to invoke Einstein you are only cheating yourself (and impact your business in that process).
What is your decision making process?
Quantifiable Uncertainty means you can express the uncertainty within some known boundaries. Like saying, “I don’t know what is the click through rate is but I am 90% confident it is between 1% and 8%”
Unquantifable Uncertainty means those factors which are unknown and cannot even be expressed in terms of odds (or probabilities).
Note: The statistical analysis shown here is based on data provided by one VC firm BlueRun Ventures. The ratings they did is likely post hoc and has biases. Hence the results are not as generic as the title says they are and have considerable uncertainties. This is also a long article and relies on linear regression and logistic regression.
Imagine you were asked to invest in ten startups. Given numerical ratings on the Team, Product, Market and Traction but knowing nothing about the specifics of the team, the exact product or the domain they play in, can you pick those that actually received a term sheet? Take this quiz and see how you do. Do not read ahead before you do the quiz.
What characteristics of a startup make it attractive for venture capitalists to invest in it? If you are a startup founder preparing for that pitch, wouldn’t it be nice to know the answer so you can prepare well to maximize your chances of getting that coveted term sheet? For those who are listening, there is no scarcity of advice. Everyone from VCs, startup founders who secured funding at significant valuations and others on the sideline, all have something to say.
Are any of these relevant to startup founders? What is noise and what is signal? Do any of these have hard numbers behind them?
Until now there was no hard quantitative data on startups that pitch to VCs and the outcome. Thanks to data from Jay Jamison, partner at BlueRun Ventures, I have data on 216 startups that pitched to his firm. Jamison rated them on four metrics, Team, Product, Market and Traction using a 5-point scale and also noted the outcome of their pitch. The outcome is rated as likelihood of getting term sheet on a five-point scale, with 5 meaning they got it.
Armed with this data we now can model if any of these traits of a startup influence its ability to get term sheet using statistical analysis. While Jamison did his initial analysis himself, it was not rigorous enough and pointed to incorrect reasons. He later shared his data with me and encouraged me to do not one but two ways of analysis this data to come up with a prediction model.
Stepwise Linear Regression
Let us say there is only one independent variable X and one outcome variable Y. Suppose we had several pairs of these, (x1,y1), (x2, y2) …. based on our observations. A linear regression model tries to find a line of the form Y = mX + C that is the best possible fit, one with least error, given the set of observations.
How good a fit is this model in explaining changes in Y is measured as ratio of two errors and is called R2 or coefficient of determination. Khan Academy has a very nice explanation of R2 that I recommend you check out. It is a positive ratio with maximum value of 1 and minimum of 0. Higher the value, better the fit.
What is that got to do with startups and venture funding? We will model the outcome, whether or not the startup got term sheet as a function of the four traits. We will build a model that has the best fit and also find how good it is in predicting the outcome.
In any regression model, if you try to model with maximum set of variables you will find a very good fit with very high R2. Such a model is useless. We want to find the minimal set of variables that we can control and also measure how the predictability of the model improves as we add variables one at a time. That is stepwise linear regression.
Step 1: Trying to model the term sheet outcome with each of the four variables, separately, I found that Team alone stands out as very good predictor with R2 of 34%. That is 34% of the changes in outcome are explained by changes in Team and 66% are not explainable by Team. It however seems to fit the commonly accepted notion that VCs invest in teams and not products.
Step 2: This step is to build yet another model that retains the Team variable from step 1 and tries to add one more from the remaining three. The second variable that has the most positive impact in improving the predictability? Market. But it did not improve the model’s predictability much. Adding Market moved the R2 only by 10%, meaning Market characteristics have very low predictability.
Step 3: You get the picture. The third variable is Traction and it did even worse with just 5% increase in R2.
Step 4: There is no step 4. The left out variable, Product, had absolutely no role to play in predicting the outcome. If you are obsessing about the product, its features and how well it compares against the others in the market, all that have no impact whatsoever in tipping VCs’ decisions. The product is not relevant.
So the only real startup characteristic with meaningful predictability for getting term sheet, using linear regression model, is how good a team you have assembled.
Now to yet another bigger surprise.
Jamison rated the term sheet outcome as likelihood on 1 to 5 scale But if you take a closer look at his intended meaning, it was really a binary coding – 5 means they gave term sheet to the startup and 1-4 means they said no in four different ways. The outcome is Yes or No. So we should not be running linear regression at all with such binary coding. The right analysis to do is to use logistic regression that measures the probability a startup with given characteristics will get term sheet. So I recoded the term sheet values as 0 and 1 and did just that
Even in this model the Product has no role to play. That should settle the argument with the product types obsessing over details.
The biggest surprise? The biggest predictor in the linear model, Team and the smallest predictor, Traction have absolutely no role in predicting the outcome. The biggest predictor with close to 80% predictability (R2 McFadden used for logistic regression) is the Market rating. The model is in fact real simple. If the market rating is 5, your startup will get funding, if not it didn’t. You play in the hottest market you get funding regardless of the other factors.
This leads to unfortunate conclusions about startups and how VCs make investment decisions.
One, money flows based on the buzz and hype. The very rating of the Market attribute is questionable. Are VCs rating the market based on true value or the prevailing hype?
Two, money flows where there is already lot of money. So more startups that play in the same hot area get funded leading to too many players in a perceived hot market resulting in many startups that are not that distinguishable from each other, fragmentation and likely too many failures.
Third, many reasonable markets with steady growth but lack the buzz, attract no funding and hence attract no startups resulting in no meaningful innovation. This likely explains the credo of Peter Thiel’s FoundersFund, “We wanted flying cars, instead we got 140 characters”.
So what is relevant to the startups? It is not really black or white. Given the investment environment and the unavoidable hype in the valley, if you want to play the game just for funding then you may do well by pitching yet another social/mobile/big data or whatever the flavor of the day is.
If you have a true meaningful innovation that is lot more than 140 characters and have a team that is unmatched in its technical expertise, you will do well by waiting to find your match.
There are many homilies on whether or not money can buy happiness. But none posed the inverse question, until now. A reasonably solid longitudinal study (performed over a period of time by following the experimental subjects) suggests happiness might lead to better income potential. It finds people who described themselves as happier while growing up went on to earn more than those who weren’t happier. Here is the PDF link.
Naturally you are asking
- What is happiness?
- How is it measured?
- When was it measured? – After all there are many studies from reputable consulting firms and academia that measure an attribute now and say all the past growth was due to that attribute.
- What about the effect of parents, more precisely genes on the earning potential?
- What are the other lurking variables here?
All very good questions and these are posed and answered by the researchers except for the last one.
They defined and measured happiness based on answers to four statements
The positive affect subscale is additively composed of the responses to the following four particular statements:
“You enjoyed life,”
“You were happy,”
“You felt hopeful about the future,” and
“You felt that you were just as good as other people.”
They did a truly longitudinal study, starting at age 16 and studying income at age 29 for a large sample size. So it avoids hindsight bias and selective recall by participants. It also avoids survivorship bias in the results.
Regarding the effect of parents, family and genes, they controlled for that by studying siblings.
Among siblings, a one-point increase in life satisfaction at age 22, compared with the mean of the family, translated into a nearly $4,000 difference in earnings at age 29, compared with the family mean.
Now that the four main questions are answered can we say categorically that it is not just mere correlation between happiness and income and happiness is the causation?
Unfortunately not. What if there is another lurking variable that led to the respondents’ happiness that is also responsible for their income potential?
But compared to most studies we see this one merits closer look.
- Apple will sell 10s of millions of iPad mini before Holidays
- iPad mini is a market share game
- There will be cannibalization but it is better to self-cannibalize
- There will be so much new volume from lower price point of iPad mini that Apple will capture marker share
- iPod Touch is a different product category and it will not be impacted by iPad Mini
My question has been centered around whether or not the new device will deliver incremental (net new). No one has done some real analysis to show what the impact is. Even my article stopped short of exact numbers. Articles by others (of course) are even worse, they expect us to believe on faith that Apple will do well with iPad mini.
Now there is some real answer, based on more rigorous analysis than just claims that self-cannibalization is better.
My analysis, using statistical modeling, shows Apple may end up selling 22-52 million iPad minis but is placing a high risk bet when it comes to profit. Let us start from the beginning.
As I did before for Pinterest revenue model I chose to do Monte Carlo analysis to find impact on Apple’s profits from iPad mini. This is a reliable tool to use when there are many variables and there is uncertainty in the result. It also helps to state the result as a probability distribution instead of absolute statements we see from some of the analysts.
The model starts with listing the different variables that feed into final result and their 90% confidence interval values. That is we list all the different variables and state the low and high values that we are 90% confident about (we are 90% confident the real value is between low and high and only 10% chance the real value is outside this range).
I am going to assume contribution margin from iPad is $225 (given its 40%-50% margin numbers stated by iSuppli and others). All volume numbers are for the full year. The trade-down numbers and the “steal” numbers come from a recent market research on iPad mini preference. Steal here means how many of current nook/Kindle/nexus customers will switch to iPad mini. New sales is the number of new customers entering the market because of iPad mini. Current iPad volume numbers are based on Apple’s past four earnings reports.
It is easy to see that
Total iPad mini sales = Trade-down volume + Steal +New sales
Profit from iPad mini = iPad Mini margin X Total iPad mini sales
Lost profit from Trade-down = iPad Margin X Trade-down volume
Net new profit = Profit from iPad mini – Lost profit from Trade-down
Note that I ignored the effect on other products both iPod Touch and iPhone.
Running the model for 1600 iterations yields some stunning results.
First the total iPad mini volume numbers. These are huge. It is almost certain that Apple will sell at least 14 million units per year. There is 95% probability that they will sell somewhere between 22 million and 52 million iPad mini. And considering all possible scenarios the expected volume is 35 million units. These kind of numbers blow out the ramp up curves we have seen with any of the electronics products.
Such numbers will bring smile to those who chase market share and will delight analysts who recommend chasing market shares. But what does that do to Apple’s profit?
Here is the big surprise. Despite huge volumes, profit estimates show Apple is playing a high risk game with iPad mini.
First there is a 47% chance Apple will lose money (not including fixed costs, just the marginal costs, so the real impact can be worse).
At its worst, there is 1% chance that Apple could see $2.2B drop in its gross profit. It does not get much better, there is 15% chance Apple could see $1 B drop in its profit.
At the other end there is only 1% chance they could make $2.3B additional profit and only 13% chance they could see $1B additional profit.
Considering all possible scenarios, the expected net new profit from iPad mini is just $97 million a year.
That is not a big enough considering other R&D and marketing expenses (fixed costs).
There you have it. Apple will likely sell 34 million units in the first year but runs the risk of seeing no impact or worse significant impact on its profit.
Analysts betting on Apple stocks, thinking iPad mini will a few dollars to their EPS, take note. iPad mini is a high risk game for Apple despite assured high volumes.