When myths triumph over metrics

Did you fill out a March Madness bracket?

I did one and I do not know any of the teams or watch any of the games. I filled out the Blindfold Bracket – one where the match-ups are presented based on offense, defense, size, experience, 3-point shooting, and hot streak metrics. I picked winner of each matchup based on a fixed algorithm I set for my self.

I probably won’t win any bracket pool. What I have is a reproducible system. You present me any match-ups with the aforementioned comparative metrics I will produce the same results repeatedly. It is a system for making decisions based on data.

Of course the data could be wrong – wrongly collected and presented or my algorithm (defense wins games) could be wrong. But those are fixable errors if the outcomes are important to me. Say the decisions are about hiring someone or making product decisions I better have a system that may start out simple but incorporates knowledge from subsequent iterations to get better over time.

Not to mention I should pay more close attention to what data is relevant, how is it collected and its quality before using it for decision making process.

Making decisions based on data does not guarantee success every time. It only gives you a reproducible and refinable system that helps you make better and informed decisions based only on metrics you set forth and devoid of any cognitive biases. The system can be simple or complex. Use as few variables or as many as possible. Could be deterministic or probabilistic. It  is far superior to decisions based on myths.

As it turns out I am not doing so well on my Blindfold Bracket. There were several upsets and prominent one is Harvard that tripped up even those with more sophisticated model to do their Brackets,

One such model is from the expert in this field, John Ezekowitz, who does this for living. His statistical model that uses additional variables like turn-over ratio and rebounding rates predicted less than 5% chance of Harvard pulling a upset and yet it did. Many such upsets happened across NCAA.

There exist a few who so far filled out near perfect bracket. They predicted all the upsets and most likely did not rely on data or models to make their decision. While far too many metrics driven Brackets are wiped out these few stand tall for their success.

What does that mean? Does reliance on metrics drive us down the wrong path? Does gut have upper hand over mind? Are all those who say, “paralysis by analysis” or  “action over analysis” correct?

Absolutely not.

Recognize the reproducibility of metrics driven approach and our ability to continuously question the data and keep refining it.

Recognize the fact that there were many thousands that filled out the bracket using their own myths and biases and just by sheer luck some are going to be correct. We cannot treat their success of those standing as triumph of gut over mind.

Recognize the fact that March Madness games are sudden-death single matchup games and not even the most sophisticated model can predict the outcome of a single matchup.

You will see similar results in business. You will see spectacular winners who shoot from their hips, went with their gut or didn’t stop to collect data. You will see their success written about in the media and social media or worse their methods touted as the recipe for success.

Go back and re-read this article. Don’t give up on metrics and methods and don’t ever pick myths over metrics.

One thought on “When myths triumph over metrics

Comments are closed.