Odds only mean so much

Last week NPR’s Morning Edition had a piece on President Obama’s poll numbers after the second debate. As their discussion turned to his chances of winning based on Intrade betting numbers,

MONTAGNE: OK, so got all those numbers. Finally, we checked the betting crowd. Intrade, which bills itself as the world’s relieving prediction market, runs an online betting service whose participants put the odds of Barack Obama winning the election at 65 percent over Mitt Romney at 35 percent.

GREENE: But, Renee, odds only mean so much. That same service said there was a 75 percent chance the U.S. Supreme Court would strike down a national healthcare law. And the court beat those odds.

Notice the bolded text and you can see the fallacy in Greene’s argument or his lack of understanding of probabilities.

First these are very uncertain events to predict and the outcome can depend on many different variables. Modeling methods like Monte-Carlo simulation and prediction markets like Intrade try to estimate the level of uncertainty (by placing a lower and upper bound on them). The net result is a probability distribution of different outcomes and in this case whether or not President Obama will  win this November.

When a model states one outcome is more likely than the other and in reality the other outcome happens one should not treat this as failure of the model. If there is no doubt who will win, there is no uncertainty, then one does not need any of these tools. By definition there is uncertainty in the outcome.  The modeling indicates one outcome (say, Obama win) is more likely than the other.

The model is doing its job perfectly well. What it really states is, “if we were to imagine million different ways of running the 2012 election, more of them show Obama win over Romney win”.

Another point is confusing results of previous model on an unrelated event with the current one. There is no relation between Supreme Court’s decision and election results.

To say, “Odds only mean so much”, as a way to dismiss all predictive models is just plain wrong.

What are the chances mom will be home when we arrive and what does this have to do with Pinterest revenue?

Update: This article will help you understand my Gigaom guest post on Pinterest revenue: How much does Pinterest actually make?

One of the games my 7 year old and I play while driving home from her school is guessing whether mom will be home when we arrive.

I ask,”what are chances mom will be home when we arrive?”
She would almost always reply, “50-50”
Not bad for someone just learning enumerating the possibilities and finding the fraction. When we arrive home there is either mom or not. So 50-50 seem reasonable.

But are the chances really 50-50? If not how would we find it?

Well let us start with some safety and feel good assumptions, my drive time is constant, there is mom, she always leaves at fixed time and she will arrive.
Other than that we need to know

  1. What time is it now?
  2. What is her mean drive time?
  3. What is the standard deviation of drive time?

Assume that the drive times are normally distributed with the stated mean and standard deviation. It is then a question of finding, in what percentage of the scenarios the drive times show an earlier arrival time. That is the probability we were looking for and it is not 50-50 simply because there are only two outcomes.

Here we did a very simple model. But who knows what the mean is let alone standard deviation. We do not. So we do the next best thing, we estimate. We do not literally estimate the mean and standard deviation but we estimate  a high and the low value such that in 90% of the cases the drive time falls in that range. Stated another way, only 10% chance the drive time is outside this range.

This is the 90% confidence interval.We are 90% confident the value is in this interval. Once we have this then it is more simple math to find the mean and standard deviation.

Mean  is average of the low and high values. 
Standard deviation is the difference between high and low divided by number of standard deviations the 90% probability corresponds to  in a standard normal curve (3.29σ).

One you have the mean and standard deviation you can do the math to find the percentage of scenarios where drive time is below certain value.

This is still simple. We treated drive time as the measurable quantity and worked with it. But drive time is likely made up of many different components, each a random variable of its own. For instance time to get out of parking lot, time to get on the highway, etc.  There is also the possibility the start time is no more fixed and it varies.

(If you want to build more realistic model you should also model my drive time as random variable with its own 90% confidence interval estimate. But let us not do that today.)

In such a case  instead of estimating the whole we estimate our 90% confidence intervals of all these parts. In fact this is a better  and preferred approach since we are estimating smaller values for which we can make better and tighter estimates than estimating total drive time.

How do we go from 90% confidence interval estimates of these component variables to the estimate of drive time? We run a Monte Carlo simulation to build the normal distribution of the drive time variable based on its component variables.

This is like imagining driving home 10,000 times.  For each iteration randomly pick a value for each one of the component variable based on their normal distribution (mean and sigma) and add them up:

drive time (iteration n) = exit time (n) + getting to highway time (n) + …

Once you have these 10,000 drive times then find what percentage of the scenarios have drive time less than certain value. That is the probability we were looking for.

From this we could say, “there is 63% chance mom will be home when we arrive”.

We could also say, “there is only 5% chance mom will arrive 30 minutes after we arrive”.

When we know there is roadwork starting on a segment we can add another delay component (based on its 90% confidence interval) and rerun the simulation.

That is the power of statistical modeling to estimate any unknowns based on our initial estimates and state our level of confidence on the final answer.

Now what does this have to do with Pinterest revenue?

Read my article in Gigaom

Sufficient but not Necessary!

The traditional media and the social media are peppered with stories on how one can achieve success like other successful entities.  Examples include, 7 habits, Good to Great, and numerous blog articles that follow the similar pattern.  Almost all of these articles look at a successful business or a person and look for observable positive traits . Then they attribute the success to the presence of such positive traits.

The general arguments against such studies include:

  1. Treating correlation as causation
  2. Different biases (survivorship, selection, availability, hindsight)
  3. Methodology errors like omitted variable bias
  4. We can’t stop because the data fit an hypothesis, data can fit any number of hypotheses.

Even if we set all these flaws aside and accept that indeed the success was the direct result of the positive traits there is another problem. These traits may be sufficient to the success but are they necessary?

Take an extreme example (for illustration). Let us say you observe a tall person in a fruit orchard. You observe her effortlessly pick much more fruits than others thanks to her height which gives her access to more opportunities. Her height was sufficient to get more fruits, but was it necessary?

Next time you see articles on “6/7/8/9 ways to do marketing/product-launches like Apple/Google/twitter/GratefulDead”, even if you look past the biases you should ask if the methods are relevant to your situation and are indeed necessary for your success.

10 Things I Try to Find Out About Surveys

I like taking surveys. Not because I prefer being a random data point that will help tip the scale on statistical significance or because I enjoy answering how likely am I to recommend the product to my friends and colleagues (on a 0-10 scale no less), it is because I see every survey as a puzzle that begs to be solved. Here is a list of what I try to find about surveys:

  1. What decision is the marketer trying to make? I am not interested in those surveys that are simply collecting data for the sake of it or selectively seeking information to add data lipstick to something they are already doing.
  2. Is the data actionable? For instance, for pricing decisions are they asking only about attitudinal willingness to pay?
  3. Is each question necessary or could they have figured out the answers without some of the questions they are asking? For instance, “How much are you paying for Microsoft services?”
  4. Have they done the necessary qualitative research and not simply cut-n-pasted a template survey?
  5. Are they finding all the information they will need with the survey? For instance,  what use in asking about school preferences if the survey did not ask if the respondent is a decision maker for the child?
  6. How likely is it  their questions will confuse other respondents? For instance, giving options like  “Never”, “Rarely”, “Seldom” and Occasionally” all for just one question.
  7. Are they sampling the right target population?
  8. Is the survey designed to find psychographic segments and not just demographics?
  9. What kind of cross-tabs and regressions will they be running on the data and how reliable will that be?
  10. Finally, are they trying to solve too many decision problems with just one survey?

What is your take on surveys?

The Long and Cross of Analytics

Anytime you see results from studies, especially from those that offer causation arguments, here is one qualifier question you should ask to determine whether or not a study is worth your time and its causation arguments  have any merit to them:

Is this a cross-sectional study or a longitudinal study?

Cross-sectional: It is the easiest one to conduct, so everyone does it. The only conclusion anyone can draw from such a study is Correlation. To find causation is simply wrong. A cross-sectional study analyzes a cross-section of the target set (be it businesses, customers etc) at a given point in time. It classifies the target set into “winners” and “losers” based on a metric the researcher chooses and looks for traits that are present in one and not the other. That is positive traits that are present in winners and lacking in losers and negative traits absent in winners and present in losers. Then it hints at or calls out the winners are winners because of their positive traits and their lack of negative traits.

Cross-sectional studies follow  the  general pattern,

“Seven/Eight/etc  habits/traits/etc of  enormously successful individuals/businesses/entrepreneurs/bloggers”

Longitudinal:  It is very hard to conduct a longitudinal study and it takes time (literally). It analyzes a target set over a period of time. It identifies winners and losers too but not based on traits but based on some action that was taken or a condition that was present at the beginning of time. Such a study follows the performance of those with the condition and those without over the period and measures the difference in their performance. Some just take point measurements at two different points of time.

Longitudinal studies follow the general pattern,

“Businesses/Entrepreneurs/Individuals who employed  factor/action/method  saw their performance increase by x%  over 7/8/9 years”

We see lot more cross-sectional studies than we see longitudinal studies.

If it is cross-sectional, I recommend you take a pass! It has all kinds of biases in flaws and very specifically confusing correlation with causation.

If it is longitudinal give it a second look.  But resist the temptation to accept the causation suggested by these studies, you still need to be aware of lurking variables and survivorship bias and most importantly beware of some that make the time flow back.

Where do you look for marketing lessons?

Today’s WSJ has an article whose theme is, “What we can learn about business from a Church?”. There are many such articles and even books that follow this theme on, “what can we learn about business, marketing, pricing, product development etc”, from completely unrelated fields (for example a street performer or a child’s Lemonade stand to which I have contributed  as well).

It is as if we think that lessons from business research, publications and successful businesses are irrelevant that we need new lessons from these unrelated wells of knowledge. May be these are indeed better sources, but I would like to caution you about these articles and studies that want to teach us:

  1. Many of these studies are cursory reviews, some just look at one individual sample. There is no rigor to the methods employed. The observer picks what is convenient and readily available to them (their neighborhood Church, lemonade stand, farmer’s market, parking lot (mea culpa)).
  2. The most common pattern is, the observer picks successful entities and look for observable positive traits. There is no attempt to study those that are not doing so well, resulting in survivorship bias.
  3. Success is defined narrowly or as an afterthought – metrics like eyeballs, sign-ups, crow etc are used. The studies do not consider alternative scenarios where success could be an order of magnitude different from the current state?
  4. There is no attempt made to look at the origins and longevity of these traits. There is one measurement made and results reported.
  5. These traits are treated as new/unique, as if these have not been reported before. The error is in not seeing the traits as examples of established marketing principles but rather as something totally new.
  6. Assuming that the traits are a result of deliberate action taken by the entity and neglecting the possibility that these could  just be random or incidental side effects.
  7. Attributing the success of the entity to the observed positive traits. That’s a causation error.
  8. Once causation is implicitly assumed, the observer makes the leap that the positive  traits are so generic (e.g., everyone should give away for free and let customers pay what they wish) that these not only apply to other entities of the same kind but also to totally unrelated entities like Tech Startups.

I believe these studies add very little value or even distract us from the main goal.  It is tempting to look for easy lessons but these so called lessons may lead us down the wrong path. Every example you see stated as a paragon of excellence should be treated as nothing more than a case study – with flaws in information reported.

Where do you look for your lessons learned?

Here are some books that will help you see the fads for they are:

  1. Hard Facts, Dangerous Half-Truths And Total Nonsense: Profiting From Evidence-Based Management by Jeffrey Pfeffer and Robert I. Sutton
  2. The Affluent Society by John Kenneth Galbraith ( chapters on Conventional Wisdom and Consumer sovereignty )
  3. Wrong: Why Experts keep Failing Us by David Freedman
  4. Fooled By Randomness By Nassim Nicholas Taleb