Did your action really make a difference?

Fooled by Randomness
Fooled by Randomness (Photo credit: Wikipedia)

We all would like to believe and assert that the action we took made a difference, or more precisely the reason for the difference, especially if we are Gurus selling snake-oil. Any positive change in the observed metric, we want to claim it all as direct result of our action. But to what level of certainty do we know that is the case?  Here is a list of questions one should ask but not many ask.

  1. Did you have a control group that is mostly identical to the treatment group (the one you chose to implement the action on)?
  2. What is the expected variance in the metric? Does it bounce around a lot over time or across Geos?
  3. What percentage of the observed change is due to that inherent variance vs. your action?
  4. At what point in the cycle did you implement the action? Was the metric unusually below its expected mean and hence was bound to turn around (regress to the mean)?
  5. What other events happened during the same period that could have contributed to the result?
  6. Are the results scalable – can it be applied across your entire business?
  7. How much more effect is left un-captured? Can you do more of the same to keep increasing results?
  8. Among all players in the market were you the only one to take the action during the period?
  9. Is the action unique to you or can others easily do the same?
  10. If everyone else took the same action what will be the net impact on your metric? How much of the benefit will remain?
  11. What other actions were available to you and how does the effect of this compare to those? What did you forego to implement this action?
  12. Finally let us question the very metric you used. Is that even the right metric you should be looking at? What if you have been measuring your success based on the most irrelevant metric?
  13. Baker’s Dozen: Did you check the dimensions of your metric?