What you can’t change, don’t measure

How many times have you heard the phrase thrown at you by anyone and their guru? No I am not asking you to keep a measure (count) of that. Whether it was a consulting firm or someone doing time and motion study that popularized it, I don’t know its origins. This statement has become a standard quip by someone, usually someone in power and position with nice title, trying to sound data driven. What that has led to is a world where we collect anything and everything that moves – from measuring sleep pattern with wearable fitness device to up to the minute update of sales pipeline.
We now have a new product positioning for this craze – Fitbit for xyz, named after the wearable pedometer that tracks the number of steps you take and the number of minutes you stay restless in your sleep. With a gauge for everything, collecting data all the time, the craziness has led to utter madness – Big Data.

So I urge you, before you put on a Fitbit on your wrist or Fitbit for customer experience, to stop and ask a few basic questions:

1. Can I change what I seek to measure?
2. Even if I can change, can I change it at the same cadence I seek to measure it?
3. If I can change, fast enough, will the change have meaningful impact on the true measures of business performance? Sure you can change the number of retweets, blog mentions, video views, “how likely to recommend on 0 to 10”, etc., but does it matter to revenue, marketing ROI, pricing effectiveness and profit?
4. Can I measure it cheaper than the effect of change?
5. Even if the answer is yes to the questions above you need to ask – what other metric I could be collecting with my limited resources?

What do you measure? I hope the answer is, “I measure what I can change to make meaningful positive impact on my business objectives”.
What you can’t change, don’t measure.

Estimating Wrong and Estimating the Wrong Metric

In business, large enterprise or startup, we make many estimates in our roles. Be it market size, addressable market,  penetration, market share growth, effect of a marketing campaign etc.

In the absence of clear data we make assumptions, look at past performance, compare to similar entities and try to estimate the metric we are interested in. In this process we make process errors – errors in spreadsheet, coding, data entry – errors that are avoidable with stricter framework or can be weeded out with a review process.

What is more dangerous are the bias driven errors that cannot be caught by any review process because we all share the same view and biases.

Here are those estimation errors that we are oblivious to. For a vivid illustration of these errors see  this story on museum visits.

Museums take pains to make the past come alive, but many are out of their depth when looking into the future.

When selling plans for a new building or a blockbuster exhibit, civic leaders and museum officials typically cite projected visitor counts. These numbers can be effective in securing funding and public support. As predictions, however, they often are less successful.

1. Estimating Exact Numbers – Estimating a single exact number with certainty because we look down on those who don’t do that as diffident.
2. Comparison Error – Making estimates based on other similar projects but err by choosing the most successful ones (not to mention what is available and recent).
3. Atypical Scenarios – Making estimates not based on what is most common and most likely to happen but on atypical scenarios.
4. Context Error – Ignoring the state of your business (stage in growth cycle etc), market, dynamics, customer segment etc.
5. Attribution Error – Attributing someone’s success to wrong traits and making our own estimates because of shared traits.
6. Using Lore – Using very large number from the past because it is part of the organization’s shared lore – never stopping to question, “Where did we get that number?”
7. Ignoring Externalities – Making estimates without regard to economic factors, disruptions, what the competitors would do
8. Underestimation –  Deliberately estimating low in order to exceed expectations

 

Finally to top all these errors, the biggest of them all is to estimate the completely irrelevant and wrong metric because it is easy —  page views, number of retweets, user base etc. So even if you fix all the above errors you end up with perfectly estimated wrong metric. I have written in detail about this here.

Do you know your estimation errors?

Looking for falsifying evidence

Here is a puzzle I saw in Gruber’s flash card for elementary school children.

More people who smoke will develop lung cancer than those who do not smoke.

What research will show smoking does not cause lung cancer?

This is not an argument about smoking, Big Tobacco, or morals. I like this question because it is simple, popular and familiar to most of us. The first statement makes us draw the most obvious conclusion – smoking causes lung cancer. Likely we wont look past this statement. And that is what makes the question very interesting.

The questions are, given all our knowledge and pre-conceived notion (so to speak), if you were asked to falsify the causation claim,

1. What research you will do?
2. What data you will seek?

This twist makes us stop, ignore our System 1 (Kahneman) and think. Finding one more example to support the claim is not difficult. Finding falsifying evidence is not only difficult but requires a different thought process.

You see numerous such causation claims in pulp-non-fiction business books (7-Habits, In Search of Excellence, Good to Great, Linchpin, Purple Cow) and blogs. Mom and apple-pie advice about startup, running a business, marketing etc. bombard us everyday in twitter. Our System 1 wants us to accept these. After all these are said by someone popular and/or in power and the advice is so appealing and familiar.

Penn Jillette of Penn and Teller wrote,

“Magic is unwilling suspension of disbelief”

For example the audience cannot ask to walk up the stage to look inside boxes. They have to accept the magician’s word for it. That is unwilling suspension of disbelief. When it comes to gross generalizations and theories supported only by the very data that is used to form them (e.g., What can we learn from xyz) we don’t have to suspend disbelief. We have the will to seek the evidence that will falsify the claim.

Do we stop and look for falsifying evidence or find solace in the comfort of such clichéd advice?

By the way, the answer to the Gruber puzzle is in looking for lurking variable. And there is none.

Does preschool lead to career success?

If you are reading this article it is highly likely your child has been in preschool or will attend preschool. But pick randomly any child from US population, you will find that only 50% chance the child goes to preschool.

The rest either stay home, where they play with parents or caregivers, or attend daycare, which may not have an educational component. Preschool isn’t mandatory, and in most places it’s not free. (Source : WSJ)

What is the observed difference in their later performance of those who attended preschool and those who didn’t?

According to Dr. Celia Ayala, research says preschool attendance points to stellar career.  She said,

“Those who go to preschool will go on to university, will have a graduate education, and their income level will radically improve,”

50% of children don’t get to attend preschool because of economic disparity. Seems only fair to democratize the opportunities for these children and provide them free preschool when their parents can’t afford them.

I do not have a stand on fairness but I have a position on the reported research and how they drew such a causation conclusion.

First I cannot make judgement on a research when someone simply says, “research says”, without producing the work, the data that went into it and the publication. Let us look at two possible ways the said research could have been conducted.

Cross-sectional Analysis – Grab a random sample of successful and unsuccessful adults and see if there is statistically significant difference in the number of those who attended preschool.  As a smart and analytically minded reader you can see the problem with cross-sectional studies. It cannot account for all different factors and confuses correlation with causation.

Longitudinal Analysis – This means studying over a period of time. Start with some preschoolers and some not in preschool and track their progress through education, college and career.  If there is statistically significant difference then you could say preschool helped. But you, the savvy reader, can see the same problems persist.  Most significantly it ignores the effect of parents – both their financial status and genes.
A parent who enrolls the child in preschool is more likely to be involved in every step of their growth. Even if you discount that, the child is simply lucky to start with smart parents.

So the research in essence is not actionable. Using it to divert resources to invest in providing preschool opportunity to those who cannot afford is not only misguided but also overlooks opportunity cost of the capital.

What if the resources could actually help shore up elementary, middle or high-school in low-income neighborhood? Or provide supplementary classes to those who are falling behind.

Failing to question the research, neglecting opportunity costs and blindly shoveling resources on moving a single metric will only result in moving the metric but with no tangible results.

Where do you stand?