Hypothesis Testing, Anecdotes and Updating Prior Knowledge

In the movie Madagascar, when  Alex, Marty, Melman and Gloria land in the island of Madagascar, their conversation goes like this:

Yeah, here we are.
Where exactly is here?
San Diego.
San Diego?
White sandy beaches, cleverly
simulated natural environment,
wide open enclosures, I'm telling
you this could be the San Diego zoo.

Complete with fake rocks.
Wow! That looks real.

A less forgiving view of Melman’s behavior will be that he started with a preconceived notion and then looked for evidence that supported his notion, ignoring those that would contradict it. Once we have made up your mind our cognitive biases nudge us to only talk to those who would support us and ask only questions that will add credence to our premise. No wonder he did not consider the fact that they were on sea shore and the San Diego zoo doesn’t open up to the sea (and the equatorial climate etc).

This is the same situation we face when we seek and use anecdotes to support our position – we seek what is convenient, available, and be selective about it.

A more accommodating view would be to assume that his statement that they were at San Diego was an hypothesis and he tested his hypothesis by collecting data. The problems I stated above regarding biases and errors in data collection apply here. So this is not a true hypothesis testing.

Same goes for situations when you talk to a few available customers and pick and choose what they say to support your case.

Even if we give it him that the data points he collected were enough and the method was rigorous, there is problem with this approach. We can’t stop when data fit one hypothesis, data can fit any number of hypotheses. If our initial hypothesis is way off and not based any any prior knowledge,  we will make the same mistake as Melman – looking for ways to make Madgascar into San Diego.

One way to reduce such errors is to start with better hypotheses – which requires qualitative research and processing prior knowledge. This is the hard part – we get paid for making better hypotheses.

Hypothesis testing, while useful in most scenarios, is still not enough. What Melman found was, given my hypothesis this is San Diego how well does the data fit the hypothesis. But what he should have asked is, “given the data, how likely is this place San Diego?”. This question cannot be answered by traditional hypothesis testing.

There is another way. For that you need to see the sequel, Escape 2 Africa. In the sequel,  similar events happen. This time they crash land in Africa as they fly from Madagascar. They end up asking the same question. This time Melman answers the same, “San Diego”, but quickly adds, “This time I am 70% sure”.

A very very generous view of this will be, he is applying Bayesian statistics to improve uncertainty in his prior knowledge using new data.  This requires us to treat the initial notion or hypothesis as not certain but as premise with uncertainty associated with it. Then we update the premise and its uncertainty as we uncover new data.

But most of the business world is not ready for it yet. If you are interested in hearing more, drop me a note.

Other articles on Hypothesis testing:

1. Sufficient but not Necessary!
2. Looking Beyond the Obvious – Gut, Mind and in Between
3. Who Makes the Hypothesis in a Hypothesis Testing?

---

Note: Of course the movie script writers were most likely not thinking about hypothesis testing let alone Bayesian statistics. The movie is used here only for illustrative purposes.

Tags: Customer Metric, Hypothesis