There is a great book titled, “Prove It Before You Promote It“, that I read a while back. It has some very sobering remarks on focus groups and how they are applied in product development.
I am reproducing in its entirety author Steve Cuno’s commentary on focus group:
Many companies hold focus groups. They fill a room with 10 to 20 carefully selected respondents and ask them questions. That much is fine. A problem occurs only when companies mistake the resultant feedback for data—and make decisions based on what they hear.
Focus groups, with an easy-to-imitate format, are a great place for incompetents to convince themselves and unsuspecting clients that they know what they’re doing. Anyone can convene a group, ask questions, and write up the answers.
I have seen focus group reports that say things like, “Seventy percent felt the packaging was too pink” or “Eighty percent said if you open a store on the West Side, they’ll shop there.” I have seen the people running the focus groups, whose role is to remain unbiased, ask leading questions like, “Would you be more or less likely to shop at a store that advertises on violent cartoons aimed at small children?”
Amazingly and sadly, businesses actually base big decisions on these groups. They make the package less pink. They open a store on the West Side. They pull their ads from Batman cartoons. And all too often they later find that consumers don’t behave the way they said they would in the focus group.
I completely agree. I have written previously about the relevance of focus groups, this book does a much better of job teaching us the pitfalls of misuse of focus groups.
What the book says about focus groups – asking a few (leading) questions and taking product decisions based on the feedback of handful of people- is very relevant to basing product and startup strategy based on the interviews with handful of customers.
When you are talking to customers, you are still forming hypotheses not testing them. The hard part is not testing the hypotheses, but forming better hypotheses to test. Focus groups and customer interviews help us make better and testable hypotheses.
If you are in marketing, run a startup, manage a product or do A/B tests you definitely should read this book.