Hypothesis Testing, Anecdotes and Updating Prior Knowledge

In the movie Madagascar, when  Alex, Marty, Melman and Gloria land in the island of Madagascar, their conversation goes like this:

Yeah, here we are.
Where exactly is here?
San Diego.
San Diego?
White sandy beaches, cleverly
simulated natural environment,
wide open enclosures, I'm telling
you this could be the San Diego zoo.

Complete with fake rocks.
Wow! That looks real.

A less forgiving view of Melman’s behavior will be that he started with a preconceived notion and then looked for evidence that supported his notion, ignoring those that would contradict it. Once we have made up your mind our cognitive biases nudge us to only talk to those who would support us and ask only questions that will add credence to our premise. No wonder he did not consider the fact that they were on sea shore and the San Diego zoo doesn’t open up to the sea (and the equatorial climate etc).

This is the same situation we face when we seek and use anecdotes to support our position – we seek what is convenient, available, and be selective about it.

A more accommodating view would be to assume that his statement that they were at San Diego was an hypothesis and he tested his hypothesis by collecting data. The problems I stated above regarding biases and errors in data collection apply here. So this is not a true hypothesis testing.

Same goes for situations when you talk to a few available customers and pick and choose what they say to support your case.

Even if we give it him that the data points he collected were enough and the method was rigorous, there is problem with this approach. We can’t stop when data fit one hypothesis, data can fit any number of hypotheses. If our initial hypothesis is way off and not based any any prior knowledge,  we will make the same mistake as Melman – looking for ways to make Madgascar into San Diego.

One way to reduce such errors is to start with better hypotheses – which requires qualitative research and processing prior knowledge. This is the hard part – we get paid for making better hypotheses.

Hypothesis testing, while useful in most scenarios, is still not enough. What Melman found was, given my hypothesis this is San Diego how well does the data fit the hypothesis. But what he should have asked is, “given the data, how likely is this place San Diego?”. This question cannot be answered by traditional hypothesis testing.

There is another way. For that you need to see the sequel, Escape 2 Africa. In the sequel,  similar events happen. This time they crash land in Africa as they fly from Madagascar. They end up asking the same question. This time Melman answers the same, “San Diego”, but quickly adds, “This time I am 70% sure”.

A very very generous view of this will be, he is applying Bayesian statistics to improve uncertainty in his prior knowledge using new data.  This requires us to treat the initial notion or hypothesis as not certain but as premise with uncertainty associated with it. Then we update the premise and its uncertainty as we uncover new data.

But most of the business world is not ready for it yet. If you are interested in hearing more, drop me a note.

Other articles on Hypothesis testing:

  1. Sufficient but not Necessary!
  2. Looking Beyond the Obvious – Gut, Mind and in Between
  3. Who Makes the Hypothesis in a Hypothesis Testing?

Note: Of course the movie script writers were most likely not thinking about hypothesis testing let alone Bayesian statistics. The movie is used here only for illustrative purposes.

Tags: Customer Metric, Hypothesis


Price as the first choice attribute or last – Pricing Page Recommendation

Take a quick look at pricing pages of most web services and products. Most offer 3 or 4 versions that differ in features, usage (number of users, responses etc) and of course price. In every pricing page I visited (sampling, not comprehensive) the first attribute is always price. Some of the pricing pages use font and other highlighting to make pricing prominent.

What if price isn’t the first attribute you present to your customers?

What if your pricing page pitches the benefits of each version before it talks about price?

What if price is the last attribute for each version listed in your pricing page?

Last week I wrote about the difference between the Price leader and Price-Less leader*. The core idea was to start the conversation with your customers about all other attributes but price. When price is not prominent, you get to talk to customers about factors that are relevant to them.

A version of the concept of Price-Less leader was published in Journal of Marketing Research Dec 2009. The  article used the term “Benefits leader” instead of  “Price-Less leader” and they made a very relevant finding,

“When customers choose benefits leader (purely based on benefits and without price information) they tend to stick with that choice even when the price information is revealed. Even when faced with a higher price, they tend to stick with their choice based on benefits”

Applying these findings to pricing page, I hypothesize, when price is listed as the last attribute:

  1. More customers will pick your higher priced versions
  2. More customers will signup for your basic version (higher conversion)

This hypothesis is based on previous research on pricing but from a different context. So it is worth testing for your pricing page before you roll out. This is definitely worth adding to the A/B testing that you probably are already doing for the rest of the pages. I recommend this A/B testing despite my earlier warnings about A/B testing.

Note that I am not recommending that you do not show the price at all or show it only after customers sign up – I am recommending that you move the price to be last attribute you list under each version.

I am every interested in hearing your results. Send me a note on your results, even if you did not find statistically significant difference.

For the analytically inclined: If you do not want to do the traditional A/B testing you can use Bayesian. But I do not recommend a full blown Bayesian verification in this case.