Hypothesis Testing, Anecdotes and Updating Prior Knowledge

In the movie Madagascar, when  Alex, Marty, Melman and Gloria land in the island of Madagascar, their conversation goes like this:

Yeah, here we are.
Where exactly is here?
San Diego.
San Diego?
White sandy beaches, cleverly
simulated natural environment,
wide open enclosures, I'm telling
you this could be the San Diego zoo.

Complete with fake rocks.
Wow! That looks real.

A less forgiving view of Melman’s behavior will be that he started with a preconceived notion and then looked for evidence that supported his notion, ignoring those that would contradict it. Once we have made up your mind our cognitive biases nudge us to only talk to those who would support us and ask only questions that will add credence to our premise. No wonder he did not consider the fact that they were on sea shore and the San Diego zoo doesn’t open up to the sea (and the equatorial climate etc).

This is the same situation we face when we seek and use anecdotes to support our position – we seek what is convenient, available, and be selective about it.

A more accommodating view would be to assume that his statement that they were at San Diego was an hypothesis and he tested his hypothesis by collecting data. The problems I stated above regarding biases and errors in data collection apply here. So this is not a true hypothesis testing.

Same goes for situations when you talk to a few available customers and pick and choose what they say to support your case.

Even if we give it him that the data points he collected were enough and the method was rigorous, there is problem with this approach. We can’t stop when data fit one hypothesis, data can fit any number of hypotheses. If our initial hypothesis is way off and not based any any prior knowledge,  we will make the same mistake as Melman – looking for ways to make Madgascar into San Diego.

One way to reduce such errors is to start with better hypotheses – which requires qualitative research and processing prior knowledge. This is the hard part – we get paid for making better hypotheses.

Hypothesis testing, while useful in most scenarios, is still not enough. What Melman found was, given my hypothesis this is San Diego how well does the data fit the hypothesis. But what he should have asked is, “given the data, how likely is this place San Diego?”. This question cannot be answered by traditional hypothesis testing.

There is another way. For that you need to see the sequel, Escape 2 Africa. In the sequel,  similar events happen. This time they crash land in Africa as they fly from Madagascar. They end up asking the same question. This time Melman answers the same, “San Diego”, but quickly adds, “This time I am 70% sure”.

A very very generous view of this will be, he is applying Bayesian statistics to improve uncertainty in his prior knowledge using new data.  This requires us to treat the initial notion or hypothesis as not certain but as premise with uncertainty associated with it. Then we update the premise and its uncertainty as we uncover new data.

But most of the business world is not ready for it yet. If you are interested in hearing more, drop me a note.

Other articles on Hypothesis testing:

  1. Sufficient but not Necessary!
  2. Looking Beyond the Obvious – Gut, Mind and in Between
  3. Who Makes the Hypothesis in a Hypothesis Testing?

Note: Of course the movie script writers were most likely not thinking about hypothesis testing let alone Bayesian statistics. The movie is used here only for illustrative purposes.

Tags: Customer Metric, Hypothesis


The Role of Information is to Reduce Uncertainty

Why do we need to do Marketing research, collect analytics,  perform A/B testing, and conduct experiments?

  1. To find out whether  the Highest paid person’s opinion (HiPPO) is true?
  2. To pick the clear winning option?
  3. To satisfy our ego that we drive decisions based on analytics?

The real purpose of  all these methods of data collection is to reduce uncertainty in our decision making. Decision making after all is about making choices. If there are no choices or you have already made your choice, then there is no real decision making.

If you have options but are not certain which one to go with, then there is uncertainty, or more precisely there is an unacceptable level of uncertainty. If it were acceptable, that is the expected results are not that different, then there is no decision making as well. Just flip a coin and go with it (My article from 2009.)

If the level of uncertainty is unacceptable,  that is choosing the wrong option will mean the difference between life and death or profit and loss – then it may be worth it to reduce this uncertainty provided the cost to get this information is less than the value differential.

Conversely, if the information you have or collect does nothing to reduce uncertainty in decision making then it is irrelevant regardless of how plentiful it is, how statistically significant it is, and how easy or cheap it is to collect it.

How do you make your decisions?

Why do you collect information?

Looking Beyond the Obvious – Gut, Mind and in Between

Update 8/14/2011: I wrote this article more than a year ago. There is a book out on the very topic of Bayesian reasoning.  Times published an article on the book. The article gives a very similar coin-toss problem. You can find the solution here.

A street smart guy called Fat Tony, a by-the-book numbers guy   called Dr.John (PhD) and Rev. Thomas Bayes walk into a bar. There they meet a man, Mr. NNT, who shows them a coin and tells them to assume  it is fair (equal probability of getting head or tail). Mr. NNT  flips the coin 99 times and gets heads each time. He then asks them,

“What are the chances of getting tails in my next toss?”

While you think about your answer, here is a background on these three people. Fat Tony and Dr.John are two imaginary people described in the book The Black Swan (pp124) by Nassim Nicholas Taleb (NNT) who describes them as follows

Fat Tony?Fat Tony’s motto is, “Finding who the sucker is”. He is remarkably gifted in getting free upgrades, getting unlisted phone numbers through his forceful charm. (Fat Tony reminds me of Soprano and his methods)

Dr. John is a painstaking, reasoned and gentle fellow who knows computers and statistics and works for an insurance company. (Dr. John reminds me of Data, with rigid rules)

There is no Thomas Bayes in NNT’s story, he died in the 18th century. His methods, however, are very relevant here.

Back to NNT’s question, which one of the two answers will you agree with?
Dr. John answers  50% because that was the assumption and each toss is independent of the other.
Fat Tony answers no more 1% and believes the coin must be loaded and it can’t be a fair game.

Dr. John follows the science of marketing by numbers to the letter. He applies hypothesis testing, sampling and  statistical significance all the time. But, he confuses assumptions with facts. When he starts with an assumption he refuses to look beyond the obvious and refine his knowledge with new data. He sticks to the prior knowledge as given and dismisses events stating otherwise.

Fat Tony has no system. He shoots from the hip. He is the big picture visionary guy.  He has been there, done that. He has no prior knowledge nor does he care about analyzing whether data fits theory.  He is simply convinced that getting heads 99 times in a row means funny business. He believes plural of anecdotes is data, worse, irrefutable evidence.

In this specific example, Fat Tony is most likely to be correct and he gets it right  not because of his superior street skills, gut feel, “Blink” but because of how he always makes decisions. He is correct in questioning the assumption but his methods are not repeatable or teachable.

Until now, I did not say what Thomas Bayes said. His answer was

“I am all but certain (almost 100%)  that the next toss will not be tails*”

There is a better way between the  street smart, gut-feel ways of Fat Tony who goes by just what what he sees (and he has seen enough) and the rigid number crunching of Dr.John who believes that the assumption holds despite data.  That’s the Bayesian way.

Bayesian Marketer  does not “assume” the very thing he is trying to prove and does not mistake statistical significance for economic significance. For a Bayesian, assumptions are just that, not irrefutable facts. He does not let the gut decide but lets it guide and effectively uses data to make more informed decisions. He knows he is making decisions under uncertainty and wisely uses experimentation and information  combined with his mind to reduce this uncertainty. (see below for the math)

That is the way to confidently pursue strategy in the presence of uncertainties!

How do you make your  decisions?

*For those mathematically inclined:

Bayesian does not look at probabilities as ratio of count of events but as a measure of certainty.  Bayesian also accounts for uncertainty and does not take the hypothesis as given (i.e., assuming the coin is fair). In this case instead of stating, ” probability of getting  heads with a fair coin is 50%”, he states his prior as, ” i am 50% confident I will get tails in a coin toss”

P(C) = 0.5   where C is for his confidence level in getting tail.
The 99 heads we saw are the data D. Bayesian asks given the observed data D, how does my estimate change. That is  he asks what is P(C|D)?

P(C|D) =  P(D|C) * P(C) / P(D)

P(D) = P(D|C)*P(C) + P(D|Not C)*P(Not C)
P(D|C) is the the chance of getting 99 heads in a row given that coin was fair. That is (1/2) raised to the power of 99, a very low number.
P(D|Not C)  is 1- P(D|C) , i.e, you can get this data in every possible scenario except with a fair coin. P(D| Not C) is almost close to 1. Here is a simpler explanation, the coin can be fair in only one way but it can be unfair in any number of ways.

Plugging in the numbers we compute  P(C|D) to be close to 0 and hence the refined estimate of confidence.

Hence his answer that says how confident he is about getting tails in the 100th toss. To reiterate unlike Fat Tony or Dr.John, Bayesian does not say what the chances are but how confident he is about the outcome.

For those raising the valid point that the coin could still be fair if you continue to toss it for 10,000 or 10 million times. Yes, you are correct and Bayesian will continue to refine his uncertainty as new information comes in. He does not stop with initial information.