Estimating Wrong and Estimating the Wrong Metric

In business, large enterprise or startup, we make many estimates in our roles. Be it market size, addressable market,  penetration, market share growth, effect of a marketing campaign etc.

In the absence of clear data we make assumptions, look at past performance, compare to similar entities and try to estimate the metric we are interested in. In this process we make process errors – errors in spreadsheet, coding, data entry – errors that are avoidable with stricter framework or can be weeded out with a review process.

What is more dangerous are the bias driven errors that cannot be caught by any review process because we all share the same view and biases.

Here are those estimation errors that we are oblivious to. For a vivid illustration of these errors see  this story on museum visits.

Museums take pains to make the past come alive, but many are out of their depth when looking into the future.

When selling plans for a new building or a blockbuster exhibit, civic leaders and museum officials typically cite projected visitor counts. These numbers can be effective in securing funding and public support. As predictions, however, they often are less successful.

  1. Estimating Exact Numbers – Estimating a single exact number with certainty because we look down on those who don’t do that as diffident.
  2. Comparison Error – Making estimates based on other similar projects but err by choosing the most successful ones (not to mention what is available and recent).
  3. Atypical Scenarios – Making estimates not based on what is most common and most likely to happen but on atypical scenarios.
  4. Context Error – Ignoring the state of your business (stage in growth cycle etc), market, dynamics, customer segment etc.
  5. Attribution Error – Attributing someone’s success to wrong traits and making our own estimates because of shared traits.
  6. Using Lore – Using very large number from the past because it is part of the organization’s shared lore – never stopping to question, “Where did we get that number?”
  7. Ignoring Externalities – Making estimates without regard to economic factors, disruptions, what the competitors would do
  8. Underestimation –  Deliberately estimating low in order to exceed expectations

 

Finally to top all these errors, the biggest of them all is to estimate the completely irrelevant and wrong metric because it is easy —  page views, number of retweets, user base etc. So even if you fix all the above errors you end up with perfectly estimated wrong metric. I have written in detail about this here.

Do you know your estimation errors?