When myths triumph over metrics

Did you fill out a March Madness bracket?

I did one and I do not know any of the teams or watch any of the games. I filled out the Blindfold Bracket – one where the match-ups are presented based on offense, defense, size, experience, 3-point shooting, and hot streak metrics. I picked winner of each matchup based on a fixed algorithm I set for my self.

I probably won’t win any bracket pool. What I have is a reproducible system. You present me any match-ups with the aforementioned comparative metrics I will produce the same results repeatedly. It is a system for making decisions based on data.

Of course the data could be wrong – wrongly collected and presented or my algorithm (defense wins games) could be wrong. But those are fixable errors if the outcomes are important to me. Say the decisions are about hiring someone or making product decisions I better have a system that may start out simple but incorporates knowledge from subsequent iterations to get better over time.

Not to mention I should pay more close attention to what data is relevant, how is it collected and its quality before using it for decision making process.

Making decisions based on data does not guarantee success every time. It only gives you a reproducible and refinable system that helps you make better and informed decisions based only on metrics you set forth and devoid of any cognitive biases. The system can be simple or complex. Use as few variables or as many as possible. Could be deterministic or probabilistic. It  is far superior to decisions based on myths.

As it turns out I am not doing so well on my Blindfold Bracket. There were several upsets and prominent one is Harvard that tripped up even those with more sophisticated model to do their Brackets,

One such model is from the expert in this field, John Ezekowitz, who does this for living. His statistical model that uses additional variables like turn-over ratio and rebounding rates predicted less than 5% chance of Harvard pulling a upset and yet it did. Many such upsets happened across NCAA.

There exist a few who so far filled out near perfect bracket. They predicted all the upsets and most likely did not rely on data or models to make their decision. While far too many metrics driven Brackets are wiped out these few stand tall for their success.

What does that mean? Does reliance on metrics drive us down the wrong path? Does gut have upper hand over mind? Are all those who say, “paralysis by analysis” or  “action over analysis” correct?

Absolutely not.

Recognize the reproducibility of metrics driven approach and our ability to continuously question the data and keep refining it.

Recognize the fact that there were many thousands that filled out the bracket using their own myths and biases and just by sheer luck some are going to be correct. We cannot treat their success of those standing as triumph of gut over mind.

Recognize the fact that March Madness games are sudden-death single matchup games and not even the most sophisticated model can predict the outcome of a single matchup.

You will see similar results in business. You will see spectacular winners who shoot from their hips, went with their gut or didn’t stop to collect data. You will see their success written about in the media and social media or worse their methods touted as the recipe for success.

Go back and re-read this article. Don’t give up on metrics and methods and don’t ever pick myths over metrics.

Can you add my one question to your survey?

No sooner you let it be known, mostly inadvertently, that you are about to send out a survey to customers than starts incessant requests (and commands) from your co-workers (and bosses) to add just one more question to it. Just one more question they have been dying to find the answer for but have not gotten around to do a survey or anything else to find the answer for.

Just one question right? What harm can it do? Sure you are not opening the floodgates and adding everyone’s question, just one question to satisfy the HiPPO?

May be I am unfair to all our colleagues. It is possible it is not them asking to add one more question, it is usually us who is tempted to add just one more question to the survey we are about to send out. If survey takers are already answering a few it can’t be that bad for them to answer one more?

The answer is yes of course it can be really bad. Resist any arm-twisting, bribing and your own temptation to add that one extra question to a carefully constructed survey. That is I am assuming you did carefully construct the survey, if not sure add them all, the answers are meaningless and in-actionable anyways.

To define what carefully constructed survey means we need to ask, “What decision are you trying to make with the data you will collect?”.

survey-processIf you do not have decisions to make, if you won’t do anything different based on the data collected or if you are committed to do whatever you are doing now and only collecting data to satisfy the itch then you are doing it absolutely wrong. And in that case yes please add that extra question from your boss for some brownie points.

So you do have decisions to make and made sure the data you seek is not available through any other channels. Then you need to develop a few hypotheses about the decision. You do that by doing the background exploratory research including customer one-on-one interviews, social media search analysis and if possible focus groups. Yes we are actually paid to make better hypothesis so you should take this step seriously.

For example your decision is how to price a software offering and your hypotheses is about value perception of certain key features and consumption models.

Once you develop a minimal set of well defined hypotheses to test, you design the survey to collect data to test those hypotheses.  Every question in your survey must serve to test one or more of the hypotheses. On the flip side you may not be able to test all your hypotheses in one survey and that is okay. But if there is a question that does not serve to test any of the hypotheses then it does not belong in that survey.Slide2

The last step is deciding the relevant target mailing list you want to send this survey to. After all there is no point is asking the right questions to wrong people.

Now you can see what adding that one extra question from your colleague does to your survey. It did not come from your decision process, does not help with your hypotheses, and most likely not relevant to the sample set you are using.

How Do You Hire A Product Manager?

You probably have seen enough job descriptions for product managers. You may have also contributed to many of them. I did too. Most of these descriptions say

The ideal candidate must be able to lead a cross-functional team through idea generation and discovery, product definition, development and product launch, sweat the details, passionate about products, etc. …

Some startups use whimsical language,

We seek only the very best
Seeking product ninja

Well job description is one thing but do we have a system to methodically evaluate candidates and make an informed hire no-hire decision?

Can we explain, with data, how we arrived at the decision?

Do we focus more on irrelevant factors or trivial factors and make wrong decisions?

Do we let observable attributes like social media presence, clever quips and any one of those metrics like Klout cloud our decision making?

Are the results repeatable? If your team evaluates similar candidates will the outcomes be similar?

What is lacking is a rigorous quantitative framework that starts with your needs and translates into repeatable decision making system. Yes you do need to start with your needs and not the attributes of someone who adore.

First your needs. Let us cut through the language plays and jargons and quantify what qualities we look for in a product manager and how important these are for the current role.

You should realize you are hiring for a basket of attributes. What is important for you is to decide upfront, before even you talk to any candidate,  how important each attribute is. Some of the attributes are (not comprehensive)

  1. Business skills
  2. Domain expertise
  3. Communication skills
  4. Technical upbringing
  5. Hustle

You may decide all of these are equally important  or one or two carry more weight than the rest. You may already have a team with deep technical knowledge and decide you are willing to de-emphasize technical knowhow of next product manager you hire. You may look at current gaps in your team and decide to bring a candidate whose attributes will address those gaps.

Whatever the needs are, you decide before you start the search process.

List out all the key attributes and decide how important each attribute is by assigning it a weight. For your convenience I have created a list of attributes and a way to assign weights.   Most important step is to stick to your weight allocation and not let yourself be distracted by any single attribute while evaluating candidates.

Next step is to define a consistent rating scale for each attribute to evaluate all candidates. Be it a scale of 1 to 6 or 1 to 100.   This is the rating scale you will use to evaluate all candidates.  The weighted sum (sum of product of attribute weight and attribute rating) is the overall score for the candidate.

Want to hire only the best? You set a realistic (and affordable) threshold to hire only those candidates that exceed that threshold, regardless of how they scored in any one attribute (Hint: it does not matter a candidate has 10000 twitter followers or remarkable presence if the overall score does not cross the threshold.)

When more than one cross the threshold you have two options. If it is just handful that make it to that level then you can apply qualitative factors among these. Or you can commit to hiring only those that scored the most. Either way you now have a rating and decision making system.

To summarize

  1. Scope your needs
  2. Assign weights to needs
  3. Define consistent rating scale
  4. Set realistic threshold for hiring criteria
  5. Set final decision making criteria

What is left is coming up with the right process and questions to evaluate candidates on each attribute.  That part has to wait, until you answer my request to take the survey.

It costs 6-7 times more to acquire new customers over retaining existing ones …

No my account was not hacked (not yet, at least). I deliberately let this commonly repeated statement be the title without qualifying it.  Of course statements like these, (this particular one made famous by Loyalty Effect) cannot stand by themselves regardless of how popular the Guru who said this is.

Let us look at this closely

  1. Let us assume this statement is true. So shall we fire our sales team, shut down all marketing spend, stop product innovations and get rid of business development?  After all this statement indicates new customers are far more expensive. Then why bother?
  2. Let us take it to the extreme. Shall we stop after the first customer?
  3. Extending this even further, say you acquired the first customer at a cost of $1 and second at the cost of $7. Then by this logic does it cost $49, $343 etc to acquire third and fourth customer?
  4. What if you are essentially in a transactional business where you really need new customers every day because the current ones won’t be there tomorrow?
  5. How do you know it is 6-7 times or only 6-7 times? What are the data and metrics used? Was it based on experimental study?
  6. How generally applicable is this to your businesses – small and large, early stage vs. mature? Is the cost the same to all businesses?
  7. What about profits from new customers, is that 6-7 times as well?

You can see how ridiculous the statement sounds now. Here is a further breakdown of problem with this retention vs. acquisition costs statement.

First it is framed around cost and does not base it on marginal benefit and opportunity cost. I also doubt that the proponents know how cost accounting is done and most likely are allocating all kinds of fixed cost share to new customers. You need to have a costing system that can correctly capture only truly incremental costs for both acquiring and retaining. Simply distributing all costs to all customers won’t cut it.

Second it suffers from sunk cost bias. The fact that you spent some money to acquire a customer in the past does not matter in the decision to do everything to retain them. If you cannot recover the acquisition cost it is sunk. You should only look at future unearned marginal profit from each customer – existing or new. At decision time of spending capital on retention vs. acquisition you need to compute the opportunity cost and truly incremental profit from each path, not encumbered by the money you have already spent on existing customers.

Third, if the cost of acquisition is indeed high don’t you think you have a marketing problem? Is it likely that you are targeting wrong customers in wrong places with wrong product, versions, messaging and prices and hence wrong low value customers are self-selecting themselves to your service? Don’t you want to spend your resources fixing this strategic problem vs. worrying about retention?

Lastly the Innovator’s Dilemma.  What if the current customers are NOT the representation of future?  By choosing to focus your resources on them instead of new customers do you lose sight of new market opportunities, how the customer needs are evolving and how their choices for the job to be done are impacted by market trends and innovations?

Does the retention vs. acquisition pronouncement sound as profound as it did before?  I hope not.

How do you make business decisions?

Pig or a Dog – Which is Smarter?: Metric, Data, Analytics and Errors

How do you determine which interview candidate to hire? How do you evaluate the candidate you decided you want to hire? (or decided you want to flush?)

How do you make a call on which group is performing better? How do you hold accountable (or explain away) bad performance in a quarter for one group vs. other?

How do you determine future revenue potential of a target company you decided you want to acquire? (or decided you don’t want to acquire)?

What metrics do you use? What data do you collect? And how do you analyze that to make a call?

Here is a summary of an episode from Fetch With Ruff Rufman, PBSKids TV show:

https://i0.wp.com/pbskids.org/fetch/i/home/fetchlogo_ruff_vm.gifRuff’s peer, Blossom the cat, informs him pigs are smarter than dogs. Not believing her and determined to prove her wrong, Ruff sends two very smart kids to test. The two kids go to a farm with a dog and a pig. They decide that time taken to traverse a maze as the metric they will use to determine who is smarter. They design three different mazes

  1. A real simple straight line  (very good choice as this will serve as baseline)
  2. A maze with turn but no dead-ends (increasing complexity)
  3. A maze with two dead-ends

Then they run three experiments, letting the animals traverse the maze one at a time and measuring the time for each run. The dog comes out ahead taking less than ten seconds in each case while the pig consistently takes more than a minute.

Let me interrupt here to say that kids did not really want Ruff to win the argument. But the data seemed to show otherwise. So one of the kid changes the definition on the fly.

“May be we should re-run the third maze experiment. If the pig remembered the dead-ends and avoids them then it will show the pig is smarter because the pig is learning”

And they do. The dog takes ~7 seconds compared to 5.6 seconds it took in the first run. The pig does it in half the time, 35 seconds, as its previous run.

They write up their results. The dog’s performance worsened while pig’s improved. So the pig clearly showed learning and the dog didn’t. The pig indeed was smarter.

We are not here to critique the kids. This is not about them. This is about us, leaders, managers and marketers who have to make such calls in our jobs. The errors we make are not that different from the ones we see in the Pigs vs. Dogs study.

Are we even aware we are making such errors? Here are five errors to watch out for in our decision making:

  1. Preconceived notion: There is a difference between a hypothesis you want to test vs. proving a preconceived notion. 

    A hypothesis is, ” Dogs are smarter than pigs”.  So is, “The social media campaign helped increase sales”. 

    A preconceived notion is, “Let us prove dogs are smarter than pigs”. So is, “let us prove that the viral video of man on horse helped increase sales”. 

  2. Using right metric:  What defines success and what better means must be defined in advance and should be relevant to the hypothesis you are testing.
    Time to traverse maze is a good metric but is that the right one to determine which animal is smart? Whether smart or not dogs have an advantage over pigs – they respond to trainer’s call and move in that direction. Pigs only respond to presence of food. That seems unfair already.
    Measuring presence of a candidate may be a good but is that the right metric for the position you are hiring for? Measuring number of views on your viral video is good but is that relevant to performance?
    It is usually bad choice to pick a single metric. You need a basket of metrics that taken together point to which option is better.
  3. Data collection: Are you collecting all the relevant data vs. collecting what is convenient and available?  If you want to prove Madagasar is San Diego then you will only look for white sandy beaches. If you stop after finding a single data point that fits your preconceived notion you will end taking $9B write down on that acquisition.
    Was it enough to test one dog and one pig to make general claim about dogs and pigs?
    Was one run of each experiment enough to provide relevant data?
  4. Changing definitions midstream: Once you decide on the hypothesis to test, metrics and experimental procedure you should stick to that for the scope of the study and not change it when it appears the results won’t go your way.
    There is nothing wrong in changing definition but you have to start over and be consistent.
  5. Analytics errors: Can you make sweeping conclusions about performance without regard to variations?
    Did the dog really worsen or the pig really improve or was it simply regression to the mean?
    Does 49ers backup quarterback really have hot-hand that justifies benching Alex Smith?What you see as sales jump from your social media campaign could easily be due to usual variations in sales performance. Did you measure whether the performance uplift is beyond the usual variations by measuring against a comparable baseline?

How do you make decisions? How do you define your metrics, collect data and do your analysis?

Note: It appears from a different controlled experiment that pigs are indeed smarter. But if they are indeed so smart how did they end up as lunch?

Segmentation – The Taboo Word

Customers, at least those who take to twitter, do not like to think of themselves as one of many in a segment. Definitely they do not want to be “targeted”. Some even want to believe that they are a segment of one, with unique preferences.

Big established companies (other than CPG/FMCG) do not like it as well. There are no issues in accepting the simple static segmentation – Geography based, customer size based, or industry type based (NACIS).  Even then, they do not use the word segmentation to describe the classification.   Anything more, like psychographic or need-based segmentation is lost on them.

Start-ups see segmentation as theory and not as something one can use the next day to move things in their business. It helps their case to believe that their product is relevant to everyone out there and creating segments seem to cut that addressable market to a fraction. They insist on using “Product-Market fit” over  “Segment – Version Fit“.

Evidence based marketing managers, pushing for segmentation are not helping their own case with how they communicate segmentation.  The fact that the method for finding customer segments rely on advanced statistical methods do not help.

For sure they do not want to rub it in on customers face that they fit a psychographic profile. Second, they should understand the mindset of the senior executives in big companies and that of the founders at startup before trying to push for segmentation.

Senior executives or startup founders, have formed an intuitive feel, right or wrong,  for what their target segment is. They may not use the word segmentation and they may not define it narrowly but it is there. So when a evidence based marketer starts rattling about the use of statistical methods to define segments and segment sizes, they risk being treated as an academic spewing “tons of theories that are not worth an ounce of action”.

The reality is, segmentation is not an academic exercise. Big or  small, companies cannot afford to cast it aside (and yes, even startups need to worry about segmentation).

How does one evangelize segmentation? Be it in a big corporation or a small startup, here is a three step process

  1. Start with the goal and not segmentation for its own sake. What is the key business challenge the decision maker is trying to solve? What do they stand to gain by finding the segmentation? Do not bother with the how until much later. Definitely do not try to highlight your statistical expertise before you sell segmentation as an absolute necessity.
  2. Position it as resource allocation problem. No business can be all things to all people. We all have only limited resources. Position segmentation exercise as the need to find the best way to allocate limited resources.
  3. Present a template of what the results of segmentation will be. Do a mock-up with plausible segment variables. Present customer profiles as defined by these and present possible segment sizes. Talk about how the business can now, “put all its wood behind an arrow”, with an attractive segment.
Once you sell the need, the how is under your control and expertise.
Startup founders, want to talk about segmentation for your start-up?