Closing lesson for 2012: Human Judgement is Often Synonymous with Bias

If there is one principle we can learn before the year ends and practice it relentlessly in 2013 and beyond, this is it.

This comes to us from this year’s Turing Award winner Dr. Judea Pearl

“Human judgement is often synonymous with bias”

As you may have noticed from hundreds of blog posts, articles and books published every year we  are bombarded with what are essentially human judgements packaged as sage advice. Be it lessons we are asked to learn from  Raj  or Steve Jobs we are presented with biased opinions of authors who cherry pick evidence to fit their preconceived notion and fail to seek contradicting evidence.

The result is all those simplistic proclamations that goad us to

Let us learn to cast aside such proclamations for they are merely biased opinions.

Note: Notice how measured Pearl is in even in this simple statement, he qualified with “often”. I am making a stronger statement here than Pearl did because of my bias and the fact that we don’t stand to lose much if we ignored most advices from gurus vs. spending time adopting them.

Pricing Strategy Vs. Pricing Parlor Tricks

A research paper published in Journal of Consumer Research, Jan 2012, found that how we present pricing affects perception

Presenting item quantity information before price (70 songs for $29) may  make the deal appear much more appealing than if the price were presented first ($29 for  70 songs).

There are many similar peer reviewed research reports that found behaviors like,

Customers are more likely to prefer prices ending with digit 9

Customers are immune to higher prices when you don’t show the $ sign

Customers pay higher prices when you write the price in words instead of numbers

Customers succumb to decoy pricing (present three options but one is asymmetrically dominated by other and hence a decoy)

Through books and TED talks these  academic reports seep into popular media and are presented as pricing lessons for businesses small and large, especially for startups. After all, these are peer reviewed research reports based on controlled experiments that found statistically significant difference, published in reputable journals and hence worthy of our trust?

May be these are true, but what do they tell us about the customers and their needs? What job is your customer hiring your product for when they pay this cleverly presented price?

The problem is these behavioral pricing tactics may just be statistical anomalies. Let me point you to a xkcd  comic that so nicely makes the point I am about to make . After what xkcd has to say, anything I say below is redundant.

Let us take the first research I quoted, “70 songs for $29 vs. $29 for 70 songs”. What could be wrong here?  Well, why specifically 70 and 29?  What other combinations did the researchers test and what are the outcomes? What about 60 for 25, 50 for 20 etc etc.

Is it possible that they had tested 20 different combinations and found that just this one produced statistically significant difference? (Like the green jelly beans in xkcd comic?). Did the researchers stash away all the experiments that produced no results and published  the one that produced this interesting result?

An opinion piece in Business Strategy Review, published by London School of Economics, pretty much says this is the case with most research we read.

The problem is that if you have collected a whole bunch of data and you don’t find anything or at least nothing really interesting and new, no journal is going to publish it.

Because journals will only publish novel, interesting findings – and therefore researchers only bother to write up seemingly intriguing counterintuitive findings – the chance that what they eventually are publishing is BS unwittingly is vast.

Pretty much we cannot trust any of the research we read.

What are likely statistical flukes get published as interesting findings on pricing and find their way into books, TED talks and blogs. The rest don’t even leave researcher’s desk. Let alone academic journal, try writing a blog post that reports, “found no statistically significant difference”. Who will read that?

What we are seeing is publication bias that is worse than any sampling bias or analysis bias and a prevalence of pricing parlor tricks presented as authoritative lessons in pricing for businesses.

When it comes to pricing your product, be it pricing cupcakes or a webapp, you would do well to look past these parlor tricks and start with the basics.

Pricing strategy starts with customer segments and their needs. You cannot serve all segments, you need to make choices. Choose the segments you can target and deliver them a product at a price they are willing to pay.

As boring and dull as it may sound, that is pricing strategy. Your business will do well to start with the most boring and dull than chasing the latest parlor trick based on selective reporting.

Everything else is distraction. May be these fine tunings have some effect but not before strategy. After you get your foundation right, then you can worry about what font to use in the sign board.

How do you set your pricing?

Other Readings:

  1. Segment-Version Fit
  2. Five Ways Startups Get Pricing Wrong
  3. Small Business Pricing
  4. Three Components of Effective Pricing
  5. Approximate Guide to Pricing Webapps  (buy access for 99 cents, pun intended )

Fail fast because successful companies failed before they succeeded

There are several versions of this statement, one way or another they glorify failures and in the name of exhorting startup founders these inspirational statements lead one to believe

  1. After a few failures success is inevitable
  2. You must fail first to succeed
  3. Fail fast so you can succeed
  4. Failures signal impending success
  5. “Failure can be a true blessing in that it educates you and prepares you for success” (from here)
  6. “Remember that most successful entrepreneurs fail good and hard before they finally make it” (same source)

All these assertions are happy to point out popular examples. The problem is the assertions are derived from the very examples they are using as evidence.

First let us make something very clear. Success and Failure are the only two possible outcomes for any venture you undertake. But the fact that there are just two outcomes does not mean they are equally probable. It is not the case of tossing a fair coin and calculating the odds of heads or tails.The chances of success and failure can be and are very different. If you take the base rate (looking at the success rate of thousands of ventures and small businesses) the success rate is 3 to 5%.

Second  even if we assume that Success and Failure are equally likely, a series of failures does not mean inevitable success. Take the coin example. The probability of getting 10 Tails in a row is same as the probability of getting 9 Tails in a row followed by a Head.

Lastly the fact that those who succeeded had failed in the past is irrelevant. Those who make such an argument pick only the success stories that are popular, recent and available to them. When you only look at those who succeeded and are still in business you are leaving all those who did finally succeed and gave up or still trying without success. Even in these cited success stories success is mostly random rather than a result of their failures. The fact that those who succeeded had “failed hard” does not mean when you fail you will succeed.

Granted they learned from their mistakes but you do not have to learn from your own mistakes.  You do not have to fail to learn. Failure is not the true blessing. Insane success with hundreds of billions of valuation even when your venture has no real product or clear value add is true blessing.

Those who advise you to fail are not being intellectually honest. Their advices are no different from those advising a gambler to bet on a slot machine that had been coming up empty for the past few hours.


Sufficient but not Necessary!

The traditional media and the social media are peppered with stories on how one can achieve success like other successful entities.  Examples include, 7 habits, Good to Great, and numerous blog articles that follow the similar pattern.  Almost all of these articles look at a successful business or a person and look for observable positive traits . Then they attribute the success to the presence of such positive traits.

The general arguments against such studies include:

  1. Treating correlation as causation
  2. Different biases (survivorship, selection, availability, hindsight)
  3. Methodology errors like omitted variable bias
  4. We can’t stop because the data fit an hypothesis, data can fit any number of hypotheses.

Even if we set all these flaws aside and accept that indeed the success was the direct result of the positive traits there is another problem. These traits may be sufficient to the success but are they necessary?

Take an extreme example (for illustration). Let us say you observe a tall person in a fruit orchard. You observe her effortlessly pick much more fruits than others thanks to her height which gives her access to more opportunities. Her height was sufficient to get more fruits, but was it necessary?

Next time you see articles on “6/7/8/9 ways to do marketing/product-launches like Apple/Google/twitter/GratefulDead”, even if you look past the biases you should ask if the methods are relevant to your situation and are indeed necessary for your success.

Where do you look for marketing lessons?

Today’s WSJ has an article whose theme is, “What we can learn about business from a Church?”. There are many such articles and even books that follow this theme on, “what can we learn about business, marketing, pricing, product development etc”, from completely unrelated fields (for example a street performer or a child’s Lemonade stand to which I have contributed  as well).

It is as if we think that lessons from business research, publications and successful businesses are irrelevant that we need new lessons from these unrelated wells of knowledge. May be these are indeed better sources, but I would like to caution you about these articles and studies that want to teach us:

  1. Many of these studies are cursory reviews, some just look at one individual sample. There is no rigor to the methods employed. The observer picks what is convenient and readily available to them (their neighborhood Church, lemonade stand, farmer’s market, parking lot (mea culpa)).
  2. The most common pattern is, the observer picks successful entities and look for observable positive traits. There is no attempt to study those that are not doing so well, resulting in survivorship bias.
  3. Success is defined narrowly or as an afterthought – metrics like eyeballs, sign-ups, crow etc are used. The studies do not consider alternative scenarios where success could be an order of magnitude different from the current state?
  4. There is no attempt made to look at the origins and longevity of these traits. There is one measurement made and results reported.
  5. These traits are treated as new/unique, as if these have not been reported before. The error is in not seeing the traits as examples of established marketing principles but rather as something totally new.
  6. Assuming that the traits are a result of deliberate action taken by the entity and neglecting the possibility that these could  just be random or incidental side effects.
  7. Attributing the success of the entity to the observed positive traits. That’s a causation error.
  8. Once causation is implicitly assumed, the observer makes the leap that the positive  traits are so generic (e.g., everyone should give away for free and let customers pay what they wish) that these not only apply to other entities of the same kind but also to totally unrelated entities like Tech Startups.

I believe these studies add very little value or even distract us from the main goal.  It is tempting to look for easy lessons but these so called lessons may lead us down the wrong path. Every example you see stated as a paragon of excellence should be treated as nothing more than a case study – with flaws in information reported.

Where do you look for your lessons learned?

Here are some books that will help you see the fads for they are:

  1. Hard Facts, Dangerous Half-Truths And Total Nonsense: Profiting From Evidence-Based Management by Jeffrey Pfeffer and Robert I. Sutton
  2. The Affluent Society by John Kenneth Galbraith ( chapters on Conventional Wisdom and Consumer sovereignty )
  3. Wrong: Why Experts keep Failing Us by David Freedman
  4. Fooled By Randomness By Nassim Nicholas Taleb

The Mind of an Entrepreneur and that of an Analyst

As I connect with and work with many entrepreneurs from my class at Haas, from Boulder, through my blog and as I read more about them,  I am compelled to make a generalization about their mindset and how they approach ideas. I see the doublespeak in my claims, basing it purely on anecdotal evidence, after all my writings on the need for evidence based management and calling out those who make such generalizations. For what it is worth, treat these as hypotheses and not as claims.

H1: It is impossible for someone who rationally estimates net present value of all options, stress tests their assumptions, meticulously conduct sensitivity analysis, determines market sizes and customer demands to start a new venture. (Dan Ariely calls this Optimism Bias, Gavin Cassar attributes this to selection bias)

Corollary: To start a venture one needs to be risk seeking and at the very least be willing to suspend their  rational mind to make the leap.

H2: Most ventures start with an entrepreneur seeing a localized problem and deciding that it needs a generic solution.

Here is  a  case study (which not should be treated as proof but rather as one of many stories that added to my conviction and pushed me to make the hypotheses).

Netflix for Hot Couture: This is a business started by two Harvard MBAs. One of they  saw the problem with women buying expensive evening wear and party wear just for single use and decided  that this problem is generic enough and needs a bigger solution. They came up with a web based service for renting designer dresses for $50 to $100. For an analyst this idea is a non-started. Look at all the issues:

  1. The idea is not unique and is easily copyable
  2. What is the problem is this addressing and what is the unique value add?
  3. What is the market size? What are the segments? What is each segment willing to pay?
  4. Fashion by definition is fickle and changes fast and is different across regions – how much inventory does one need and how big is the risk of carrying this inventory?
  5. For a $1000 dress can we rent it out enough times to not only cover the costs but also turn profit?

These alone are enough reasons to not starting this venture but not to the two people who not only started this but have successfully found funding for it. Will this venture go big? My analytical part says no, but as a fellow human I wish  them well and hope they will go big and succeed.

On a side note, whoever described this idea as “Netflix for Hot Coutre” knows how to do marketing and following the advice of CEO Mr. Marc Benioff. This metaphor helps with explaining the new service but one should not reduce Netflix to just a DVD rental service, you should read the vision of Netflix CEO on what Netflix is about.

A Netflix Model for Haute Couture