Can you add my one question to your survey?

No sooner you let it be known, mostly inadvertently, that you are about to send out a survey to customers than starts incessant requests (and commands) from your co-workers (and bosses) to add just one more question to it. Just one more question they have been dying to find the answer for but have not gotten around to do a survey or anything else to find the answer for.

Just one question right? What harm can it do? Sure you are not opening the floodgates and adding everyone’s question, just one question to satisfy the HiPPO?

May be I am unfair to all our colleagues. It is possible it is not them asking to add one more question, it is usually us who is tempted to add just one more question to the survey we are about to send out. If survey takers are already answering a few it can’t be that bad for them to answer one more?

The answer is yes of course it can be really bad. Resist any arm-twisting, bribing and your own temptation to add that one extra question to a carefully constructed survey. That is I am assuming you did carefully construct the survey, if not sure add them all, the answers are meaningless and in-actionable anyways.

To define what carefully constructed survey means we need to ask, “What decision are you trying to make with the data you will collect?”.

If you do not have decisions to make, if you won’t do anything different based on the data collected or if you are committed to do whatever you are doing now and only collecting data to satisfy the itch then you are doing it absolutely wrong. And in that case yes please add that extra question from your boss for some brownie points.

So you do have decisions to make and made sure the data you seek is not available through any other channels. Then you need to develop a few hypotheses about the decision. You do that by doing the background exploratory research including customer one-on-one interviews, social media search analysis and if possible focus groups. Yes we are actually paid to make better hypothesis so you should take this step seriously.

For example your decision is how to price a software offering and your hypotheses is about value perception of certain key features and consumption models.

Once you develop a minimal set of well defined hypotheses to test, you design the survey to collect data to test those hypotheses.  Every question in your survey must serve to test one or more of the hypotheses. On the flip side you may not be able to test all your hypotheses in one survey and that is okay. But if there is a question that does not serve to test any of the hypotheses then it does not belong in that survey.

The last step is deciding the relevant target mailing list you want to send this survey to. After all there is no point is asking the right questions to wrong people.

Now you can see what adding that one extra question from your colleague does to your survey. It did not come from your decision process, does not help with your hypotheses, and most likely not relevant to the sample set you are using.

The two flows – Established Enterprises vs. Entrepreneurial Ventures

See the full research here.

In finding the first customer within their immediate vicinity, whether within their
geographic vicinity, within their social network, or within their area of professional expertise, entrepreneurs do not tie themselves to any theorized or pre-conceived “market” or strategic universe for their idea. Instead, they open themselves to surprises as to which market or markets they will eventually end up building their business in or even which new markets  they will end up creating.

While a traditional (read established) business start with well defined markets, segments, targeting and product positioning to reach end customers entrepreneurial ventures start with single customers they have and move up to market definitions and sometimes creating new markets in that process. Of course, once they reach market definition the ventures now become established enterprises and revert to the first flow for their decision making.

The problem is the risk involved and how many of the startups actually move past each stage. The fact that some have made it does not mean any startup can succeed by starting with few available customers, identify more and move to define an entire market. What it really means is, as many startups try their hypotheses, testing different customers, a few will eventually traverse the path to define the market.

Estimating iPad mini Cannibalization

Today there was a survey out from Cowan and Co that got written about in almost every blog. It is about the effect of iPad mini on iPad sales. Most quickly jumped to the obvious conclusion (in articles with catch headlines too) that iPad mini will go on to add significantly to Apple’s profit with “inconsequential” cannibalization of iPad.

Here is my representation of the survey results using the data from Cowan and Co and let me tell about the problems with this survey and hence the conclusions.

First this is a stated preference study measuring the attitude of the customer and not the actual behavior. It is well established in marketing research literature that stated preference surveys overestimate behavior at point of purchase.

Second the survey question was specific to iPad mini, asking them specifically if they would buy iPad mini in the next 18 months. They did not ask them about other tablets in their consideration set nor did they ask them if they planned to purchase Kindle Fire, Nexus or nook. That is too narrow, anchors them on single choice and ignores other possibilities. Had they asked, “Which tablet will you buy?” and reported percentage distribution of different tablets it would have been much better.

Third, they slice and dice the 24 samples who reported switching from another device to report cannibalization and conversion from other tablets. The number 24 is too small to make any meaningful estimate, especially when 8 report switching from iPad and 3 report switching from Fire. If these were accurate estimate then it also point to even smaller impact on Kindle and other tablets.

Fourth, a more significant problem with the question  ”What device will iPad mini replace?” is it is just plain wrong as respondents were not primed to compare price and value of each. One right way is to do a (choice based) conjoint analysis to find the respective share of different tablets – iPad, iPad mini, Fire, Nexus etc.

Fifth, let us take the 6.1% new buyers number at face value. You can interpret this as 6.1% of all those who would buy a tablet would choose iPad mini. That is, iPad mini is not bringing in many new buyers into the market. Had they asked what tablets would they buy this number would likely pale in comparison to others. So it is overreaching to say, “its low-price will bring in new customers”.

Netting it out, there is not enough validity in the data to make bold predictions about iPad mini. There are indeed many uncertainties and those are not considered let alone quantified by this study. What you have is someone’s wishful thinking supported with non-scientific sampling and analysis.

For the record. here is my statistical analysis on iPad mini numbers and incremental profit it would drive.

4 Ways You Can Put Google Customer Surveys To Work Today

As I previously wrote, Google Customer Surveys is a true business model innovation. It helps publishers unlock value from their digital assets and enables market researchers reach new audience they otherwise would not have found. I expressed my reservations on their positioning in my previous article

But I do not get what they mean by, “look for correlations between questions” and definitely don’t get, “pull out hypotheses”. It is us, the decision makers,who make the hypothesis in the hypothesis testing. We are paid to make better hypotheses that are worthy of testing.

Since I wrote that article, their Product Manager emailed to say they removed their statement on, “pull out hypothesis”.

This is a limited tool with ability to ask just one question and no way to ensure that the same user will answer multiple questions for doing customer level analysis.

There is one more item which is their minimum sample size. You cannot order anything less than 1000 samples.

Despite these reservations I see Google Customer Surveys as an effective tool for product/brand managers, researchers and small businesses for these purposes:

1. Aided Recall:  Present them a choice of different brands ask them how many of these they recognize.
When you are trying to get very quick and high level data on customer awareness or preference of your brand, this is a great tool. The results are especially actionable when you get extreme results like no one knows about you.
If you are trying to find which brand they recognize the most then you can do that as well with different question type. However, due to its question format limitation, Google Customer Surveys cannot help with Unaided recall.

2. Finding Consideration Set: Present them a choice of different brands and ask them how many will they consider buying for solving a particular need. This is similar to Aided Recall but the question is more focused. You are not simply asking about awareness but whether your brand makes it into their consideration set.

3. Brand Association: Present them an image or a statement and ask them to pick a tag-line or brand they believe goes with it. Another variation of this question is asking them to associate your brand with an unrelated field. A typical example is, “if our brand were a movie actor, who will it be”.

Ability to use images is a very powerful feature. It creates many different opportunities. For example for testing your advertising copy or the images you use in your collateral. It is better to poll your audience whether the image you used looks more like a bean bag or boxing glove before you launch your expensive advertising campaign.

4. Consumer Behavior Research: This is a whole class of hypothesis testing you can do with Google Customer Surveys. While it is not a tool for A/B split testing, you can use it test your hypothesis on customer preferences or their susceptibility to anchors and other nudges. Before collecting results you need to specify a reasonable hypothesis that is worth testing. When you collect data you can test for statistical significance using Chi-square test to validate your hypothesis. Do keep in mind that sometimes data can fit more than one hypotheses

There is however a big limitation because of the length of questions you can ask (as you see in the third option in the image on the left).

There you have it. A tool with limitations but is effective for specific areas. It opens up new ways to collect data and test when none existed before.

A corollary for this post would be cases where you should not use this tool. That includes finding price customers are willing to pay or asking them about how important a single feature is. You have to wait for another post for the reasons.

Google Customer Surveys – True Business Model Innovation, But

Summary:Great business model innovation that points to the future of unbundled pricing. But is Google customer survey an effective marketing research tool? Do not cancel SurveyGizmo subscription yet.

Google’s new service, Customer Surveys, is truly a business model innovation. It unlocks value by creating a three sided market:

1. Content creators who want to monetize their content in an unbundled fashion (charge per article, charge per access etc)
2. Readers who want access to paid content without having to subscribe for entire content or muddle through micro-payments (pay per access)
3. Brands seeking customer insights, willing to pay for it but have been unable to find a reliable or cheaper way to get this

When readers want to access premium content they can get it by answering a question posed by one of the brands instead of paying for access. Brands create surveys using Google customer surveys and pay per use input.

Google charges brands 10 cents per response, pays 5 cents to the content creators and keeps the rest for enabling this three sided market.

Business model is nothing but value creation and value capture. Business model innovation means innovation in value creation, capture or both. By adding a third side with its own value creation and capture Google has created an innovative three way exchange to orchestrate the business model.
This also addresses the problem with unbundled pricing, mostly operational challenges with micro-payments and metering.

But I cannot help but notice severe sloppiness in their product and messaging.

Sample Size recommendation: Google recommends brands to sign up for 1500 responses. Their reason, “recommended for statistical significance”.
Statistical significance has no meaning for surveys unless you are doing hypothesis testing. When brands are trying to find out which diaper bag feature is important, they are not doing hypothesis testing.

What they likely mean is Confidence Interval (or margin of error at a certain confidence level). What is the margin of error, at 95% confidence level? With 1500 samples, assuming 200 million as the population size it is 2.5%. But you do not need that precise value given you already have sampling bias by opting for Google Customer Surveys. Most would do well with just 5% margin of error which requires only 385 responses or 10% which requires only 97 responses.

Recommending 1500 responses is at best a deliberate pricing anchor, at worst an error.

If they really mean hypothesis testing, one can use a survey tool for that, but it is not coming through in the rest of their messaging which is all about response collection. The 1500 responses suggestion is still questionable. For most statistical hypothesis testing 385 samples are enough (Rethinking Data Analysis published in the International Journal of Marketing Research, Vol 52, Issue 1).

Survey of one question at a time: Brands can create surveys that have multiple questions in them but respondents will only see one question at any given time.
Google says,

With Google Consumer Surveys, you can run multi-question surveys by asking people one question at a time. This results in higher response rates (~40% compared with an industry standard of 0.1 – 2%) and more accurate answers.

It is not a fair comparison regarding response rate. Besides we cannot ignore the fact that the response may be just a mindless mouse click by the reader anxious to get to their article. For the same reason they cannot claim , “more accurate”.

Do not cancel your SurveyGizmo subscription yet. There is a reason why marketing researchers carefully craft a multiple question survey. They want to get responses on a per user basis, run factor analysis, segment the data using cluster analysis or run some regression analysis between survey variables.

Google says,

The system will automatically look for correlations between questions and pull out hypotheses.

I am willing to believe there is a way for them to “collate” (not correlate as they say) the responses to multiple questions of same survey by each user and present as one unified response set. If you can string together responses to multiple questions on a per user basis you can do all the statistical analysis I mentioned above.<;

But I do not get what they mean by, “look for correlations between questions” and definitely don’t get, “pull out hypotheses”. It is us, the decision makers,who make the hypothesis in the hypothesis testing. We are paid to make better hypotheses that are worthy of testing.

If we accept the phrase, “pull out hypotheses”, to be true then it really means we need yet another data collection process (from a completely different source) to test the hypotheses they pulled out for us. Because you cannot use the very data you used to form a hypothesis to test it as well.

Net-Net, an elegant business model innovation with severe execution errors.

One right price is better than three wrong prices: SurveyGizmo Simplifies Pricing

This post is my interview with CEO of SurveyGizmo, Christian Vanek on their pricing strategy.

A few weeks back I wrote about the continuing changes to SurveyGizmo pricing. It turned out they have been A/B testing their pricing for a while and I had slipped through the crack, finding both the offers. Last week I sat down (over phone) with SurveyGizmo CEO, Christian Vanek and their web marketing lead Kipp Chambers for a conversation on their new pricing.  Christian happily shared with me  the genesis and details of this simplified pricing.

The details are sure to add new dimension to the thinking of most startups that see pricing as simple freemium model or do it as tactical afterthought. Their analytical process, understanding of customer mix and their willingness to go against the conventional wisdom are exceptional traits that need to be commended.

Pricing is lot more than an eye-candy pricing page!

What was their pricing before the change?

Take a look at their previous pricing page. Their pricing options and the pricing page design look not much different from numerous other webapps out there. In fact there are wordpress templates available to show this classic three column design with the “suggested version” highlighted.

One glaring difference is, while most webapps include their free version as one of the three presented, SurveyGizmo showed their free version as a footnote.
Otherwise this is nothing more than a  instance of what Hal Varian described as Goldilocks pricing.

What is the change?

Gone are the multiple editions and the pricing page eye candy to nudge customers to a specific edition. There is just one edition with all the features including the advanced features that used to be available only in the higher priced versions. Most importantly, they used to limit the number of responses per month and now they eliminated that limit as well.

In the past they had a cheaper $19 plan even though it was not prominently featured in the pricing page. Now that is gone along with the $159 Enterprise Plan that was prominently featured and highlighted in the middle of the pricing page.

After this pruning, all is left is just one version – no name  for it (like the new iPad)- offered at $50 for the first user and a flat fee of $20 per additional user.
Another point to note is there is no non-linear pricing built into the price list. Whether it is 100 additional user of 1 additional user, the price is the same, $20 per additional user.

To discuss this change, the drivers behind it and how they arrived at it, I talked to SurveyGizmo’s Christian Vanek, their CEO, and Kipp Chambers. Here is what they had to say.

Why are you open to sharing this information? Isn’t pricing strategy meant to add to your competitive advantage?

“We have a company policy of no secrets”, said Vanek. He stayed true to this policy even when I later asked him about SurveyGizmo’s future product roadmap.  ”Regarding SurveyGizmo’s pricing there is nothing really to be protective about. As soon as  the pricing page went up our competitors likely saw it. Or they will know when your article goes up. Even before this, people were copying the pricing plans and the pricing page down to the name of our plans and their feature set. Once they had comparable plans they were competing on price”. Vanek adds he could either spend all his energy protecting ideas or spend his energy on better execution and coming up with newer ideas. The choice is clear to him.

What are the drivers for this major pricing change?

We had our $19 plan, the $49 plan and the $159 plan. We found several key things from our analysis of our customers.

1. Very people were opting for the $19 plan. Some of those who chose it for price realized they did not have all the features they needed and were calling us about that. In most cases we ended up enabling the additional features for them. We are not going to tell our customer, ‘you need to pay additional just for that feature’. Some upgraded to higher priced plan just for a brief period to use the advanced features and downgraded right away when their job was done.
2. Those who picked the $159 plan were using only 10% of all possible features they get with it. We were taking lot more money from our customers who were not taking full advantage of what they were paying for.
3. What if a customer wants only one of the feature offered in higher priced version and that is the only one they want? Why should they pay more just for that? We tried for a time some kind of a la carte pricing but it was not the best of experience for our customers.
4. Surprisingly, customer satisfaction was low among those who chose the lowest priced plan and high among those who chose the higher priced plans. You could argue this is because their purchasing decision itself may have something to do with satisfaction rating.

Considering all these we thought, there is really only plan that served customer needs and presenting three options is likely aggravating customer choice by adding to their cognitive costs. So we decided to test this hypothesis.

This is so different from what every other webapp startup is doing.

Presenting  three plans, any three plans, at different price points and hoping customer will pick the one they want is shotgun approach to customer segmentation. It came apparent to us to retire the shotgun and get sniper”. (Vanek calls this his Call of Duty metaphor, “almost any business lesson can be learned from Call of Duty”, and adds The Lord of The Rings after my prompting*. )

“I think we are seeing now the end of the freemium model, signing up for free and then trying to up-sell. Our value is in providing both a great product and great service to go with it to customers who need and value our product”.

So you are giving up those customers who are willing to pay $20?

These customers were never ours to begin with. Customers who want free survey or want to pay $10 or $20 a month have always been SurveyMonkey’s customers. We are okay with that. If a customer is happy with a competitor we are okay with that. These were the customers who anyway ended up getting the features from higher priced plan because we did not want to say to them, that is extra.

What about profits lost by eliminating $159 plan?

“This was our fear as well and we discussed this internally. It would seem silly to give up on the higher priced plan. In essence you have to bring in 3 new customers at $50 level for every $159 customer we are giving up by eliminating this plan. We asked internally, can we do this? Happy to say we are doing very well after we moved to single price plan.”

“When we discuss our features with customers showing them how we compare feature for feature with competitors and then show them the price, they ask, ‘okay, why such a low price? What is the catch?’. There is no catch. We don’t have to overcharge for the product.”

About the change process?

“We did lots of A/B testing. We found that customer decision was easier with just one pricing option. In fact when we presented the simplified plan in split testing  that charged $50 for first user and  $20 for each additional user we found customers were signing up more than one user than they did with three pricing options.  We are serving marketing research field, we should be doing our homework before such change. Only after a lengthy testing process and data analysis we decided to go with this change.”

It is acceptable for a pricing geek like myself to say cognitive cost, how is that you are thinking about it?

For this Vanek seems to believe this is common sense. A customer who has to weigh multiple plans, the features it has and the price points suffers significant cognitive cost. “We work with lots of researchers who work on cognitive research and we understand the cost to customer from choice.”

Final words?

By eliminating the three plans and going to a single plan we have narrowed the field. We are targeting only those customers who want and value the advanced features.

---

*Talking of The Lord of The Rings, Vanek says his super power is he has the voice of Saruman.