The Real Limitations of Evidence Based Marketing

This is not a reply to Seth Godin’s post. While his post triggered this, the usage of the term Marketing is different between the two articles. To me, Mr. Godin’s article’s use of marketing reads more about marketing communication and messaging. I cannot agree more on the need to present the message in more interesting and acceptable format than just gobs of facts and data. We cannot hope that the rational automatons will figure it out, years of behavioral economics research has shown that we are anything but rational.

Marketing, to me,  means strategic marketing – activities that come way before we get to messaging. Like choosing the segment to target, the product versions to deliver, the pricing strategy and how to reach these customers.  This article is about that definition of marketing. However, Mr. Godin’s use of the phrase “Evidence Based Marketing” for a messaging tactic will create confusion in the minds of many. Using data and facts in messaging is a tactic, it is not Evidence Based Management.
Read on.

Evidence based marketing is flawed, it is rife with multiple errors. The common error types are

Inherent Data Errors: Data is noisy and incomplete. Sometimes the readings can be just plain wrong due to probabilistic nature of data source.
Environmental Errors: Context and environment collude to further muddy the water by tricking our eyes, minds and tools. Due to no fault of our own we see mountains when there aren’t any (like John Ross’ Croker Mountains).
Observer Effect: Call it Heisenberg uncertainty principle or Hawthorne Effect. The very act of reading the data can taint it.
Methodological Errors: Call it sampling error or survey error, our methods of data collection are inadequate. They introduce their own errors that are sometimes unquantifiable.
Data Collection Errors: We do not know what or how to measure. We ask the wrong survey question or add the wrong stimulus  but treat the received data as answer to the question we had in mind.
Analysis Errors: Test for statistical significance is flawed and misused. Not just Type-I and II errors, the very method of hypothesis testing is flawed. We collect data conditioned on the hypothesis being true and declare, “since data fit the hypothesis we conditioned to be true, the hypothesis is true”. We take comfort, wrongly, in using lower p-values. When data fit the hypothesis we stop, ignoring the fact that data can fit any number of hypotheses.
More Analysis Errors: We take comfort in adopting more rigorous and computationally intensive analytical methods. We treat correlation as causation and even regression as causation, look to the past to guide the future. We ignore lurking variables that could explain common causation.

Yet, Evidence Based Marketing is preferable to the alternative – based on guts, fads, Guru’s revelation, opinion of the person with highest title, and just plain observations.

The alternative is rife with errors that are not easy to see or measure – Cognitive errors. We suffer from multiple cognitive biases like Recency Effect, Availability Bias, Selection Bias and Survivorship Bias.Worse, the alternative rely on Commitment Bias and Agreement Bias (peer pressure) to push through a version of truth. Here the truths are unfalsifiable and it is forbidden to look for falsifiable evidence. Anyone stepping outside the accepted norm is labeled and shamed into compliance.

Evidence Based Marketing is flawed but it is clear and explicit about the flaws and lets the decision maker be the judge. The good practitioners know its flaws and bad ones cannot overlook it for too long.

Evidence Based Marketing eliminates the need for a charismatic leader and survives across leadership changes. The Alternative relies on the words and presence of the leader and the methods change when leadership changes. One myth is replaced by another.Both Evidence Based Marketing and its Alternative will take us to Madagascar but will try to tell us it is actually San Diego. Evidence Based Marketing would do that with data selection error, Alternative will just tell with authority.

When  uncovering new data shows otherwise, Evidence Based Marketing will reduce its certainty and state the likelihood that its prediction is correct.

Evidence Based Marketing is about forming a repeatable and falsifiable theory. The practitioners know that their is theory is true only because there isn’t data to prove it otherwise.

Which pill is it going to be for you?