10 Problems with Foursquare Big Data Prediction of iPhone 6s Sales

Foursquare, an App that tracks your movements and checkins, made a prediction on launch weekend iPhone 6s sales. It is a good PR to latch on to a major event to showcase their new entry into data analysis. This is also going to be used as an example of Big Data analysis. Since Foursquare has access to time series data on locations of its users, it is argued that they can put that data to work to “unlock value from it”.  Combine this movement with location of stores, you have foot traffic information on stores which they believe is a powerful dataset.

Here is how Foursquare made the prediction,

So our Place Insights team looked into foot traffic at Apple stores leading up to the launch of the iPhone 5, 5S, and 6 and analyzed it alongside Apple’s public sales data.

Combining Foursquare’s foot traffic data with Apple’s sales data on a graph shows how closely the two are linked. Visit growth is clearly a strong general advance indicator for sales performance of the launch weekend.

Foursquare predicts that launch day foot traffic will be about 360% of a typical Friday. This likely means that Apple will sell 13–15 million iPhones this weekend,

But is the method valid? Is the prediction trustable? The answer is NO to both questions. This is yet another case of chasing Big Data and letting spurious correlations make seemingly correct predictions. Here are the problems with Foursquare’s method and prediction:

1. Does foot traffic mean sales? Isn’t it possible more simply visit the store during launch weekend to look at the new phones?
2. What percentage of people who walk into Apple stores have Foursquare app? Is that a representative sample.
3. What other factors will drive foot traffic? This time there is iPad Pro which is going to bring in more people who may want to see it in action.
4. Problem with iPhone 5s data used by Foursquare. iPhone 5s was not offered for preorder and did not put up big numbers in its initial quarter.
5. What percentage of iPhone sales happen in Apple stores? Even if we accept point 1 above we should scope the effect of Apple stores on total sales. According to a 2012 survey,  Apple Stores accounted for just 21% of the iPhone Sales. Even if this is different for opening weekend that explains only portion of sales.
6. More sales are shifting online. Apple said the online demand for iPhone 6s Plus has been exceptionally strong and exceeded their own forecasts for the preorder period.
7. More sales happening though other channels. T-Mobile CEO stated iPhone 6s pre-orders on T-Mobile are up an impressive 30% compared to iPhone 6 pre-orders last year.
8. Changes in pricing model affect sales. This year subscription pricing for iPhone 6s is available from all carriers and even from Apple. There is a price war on monthly subscription fee between T-Mobile and Sprint. These factors drive new customers that is going to increase launch weekend volume.
9. China. China China. Last two times Apple did not launch in China at the same time. iPhone 6s was available in China at the same time as US.
10. Finally there is no information on confidence level or likelihood information in the prediction. I give them credit for predicting a range vs. one absolute number. But they do not state at what confidence level. That is the difference between big data analysis vs. statistical analysis.

What we have here is a faulty method that falls apart despite Foursquare’s access to largest database of information on the foot traffic of people around the globe. All that big data does not help them see Madagascar from San Diego.

Could the rest of us make better predictions without the Big Data? Yes we can if we do scenario analysis with the new factors that affect iPhone 6s sales. And we can state not only a range but also at what confidence level.

Big Data has applications but this is not it. Foursquare has bigger ambitions to monetize all their big data by making similar predictions for Marketing, Real Estate, Finance and Credit Scoring. All these predictions will suffer from the same challenges unless you build a more comprehensive model with foot traffic data as just one part of prediction. Otherwise we will be looking for Shamu in Madagascar.

You’re stuck in an elevator with someone who read a tweet about a study on women in boards

Let me start by repeating what I wrote a while back on the faulty analysis by Credit-Suisse,

Here is an undeniable fact – considering ~50% of the population is women and 50% of the board of directors are not women, we can safely say women are underrepresented in corporate boards. Now let us return to the reported study in question that makes a faulty case for adding women board members. My arguments are only with errors in the research methods and its application of faulty logic. Nothing more.

I will add to this my case against those who keep quoting such studies as evidence for their side.

The title of this article is inspired by Adrienne LaFrance. She wrote a blog post titled , “You’re stuck on an elevator with someone who loved that Sarah Lacy article. Now what?”

In that article Ms. LaFrance makes point by point argument to what seem to be silly questions from a clueless elevator companion who fell in love with Ms.Lacy’s post.  For one such question Ms.LaFrance quotes as evidence the research by Thomson-Reuters,

You should check out this study from earlier this year that showed how diverse corporate boards outperform those with no women. You’d think that a company like Twitter would put its business interests first.

She isn’t alone in quoting this study, almost everyone taking   Mr. Vivek Wadhwa‘s side use this study. I am not sure how many read this report or looked at its methods and caveats. Let me do that in this article.

Here is the link to the said research report.

1. Does the board matter?: The study starts with unverified assumption that a company’s board matters to its performance and then goes on to see differences in performance between boards. If your hidden hypothesis you took for granted is false it does not matter what your stated hypothesis is.
   What the study does is, If A=TRUE,  then A(With Women) > A(Without Women)
   You can see that if A =FALSE, the rest does not matter.You might want to stop here as nothing else matters after this error.
2. Control Variable: When you want to study the influence of a single variable you want to make sure all other variables are held constant. But when you read this report it is clear that they have no way to do that. They started with composition of a company board in 2007 then compared the performance of a group of companies over a period. There are two many uncontrolled variables during this period  (tech trends, market trends, industry verticals, etc.) and these affected different companies differently.
3. Error in Comparing Averages?: The comparisons are done on averages. There is a group of companies with mixed board and then there is another without women in board. The two groups are compared against another group, the benchmark which consist of companies of both kinds.

   The report says companies with women on board did marginally better or same as the benchmark while those with no women on board did 10-15% lower than the benchmark.

   First you notice that the difference in performance is not as significant s those who quote the study. Next you want to ask a simple clarifying question here.  If the benchmark has both types of companies, if one subset is  underperforming by 10-15%, shouldn’t the other subset outperform by 10-15% to bring it back to benchmark average?

   The only explanation I can think of is average hides details here. There must be a few companies in each side that are significantly different from the arithmetic mean for that group and they account for the difference. If you leave out these samples and compare again, the difference will likely vanish.

Now to the question of what to do when stuck in an elevator with someone who merely heard about the Thomson Reuter’s study?

Just smile and nod.

 

Does preschool lead to career success?

If you are reading this article it is highly likely your child has been in preschool or will attend preschool. But pick randomly any child from US population, you will find that only 50% chance the child goes to preschool.

The rest either stay home, where they play with parents or caregivers, or attend daycare, which may not have an educational component. Preschool isn’t mandatory, and in most places it’s not free. (Source : WSJ)

What is the observed difference in their later performance of those who attended preschool and those who didn’t?

According to Dr. Celia Ayala, research says preschool attendance points to stellar career.  She said,

“Those who go to preschool will go on to university, will have a graduate education, and their income level will radically improve,”

50% of children don’t get to attend preschool because of economic disparity. Seems only fair to democratize the opportunities for these children and provide them free preschool when their parents can’t afford them.

I do not have a stand on fairness but I have a position on the reported research and how they drew such a causation conclusion.

First I cannot make judgement on a research when someone simply says, “research says”, without producing the work, the data that went into it and the publication. Let us look at two possible ways the said research could have been conducted.

Cross-sectional Analysis – Grab a random sample of successful and unsuccessful adults and see if there is statistically significant difference in the number of those who attended preschool.  As a smart and analytically minded reader you can see the problem with cross-sectional studies. It cannot account for all different factors and confuses correlation with causation.

Longitudinal Analysis – This means studying over a period of time. Start with some preschoolers and some not in preschool and track their progress through education, college and career.  If there is statistically significant difference then you could say preschool helped. But you, the savvy reader, can see the same problems persist.  Most significantly it ignores the effect of parents – both their financial status and genes.
A parent who enrolls the child in preschool is more likely to be involved in every step of their growth. Even if you discount that, the child is simply lucky to start with smart parents.

So the research in essence is not actionable. Using it to divert resources to invest in providing preschool opportunity to those who cannot afford is not only misguided but also overlooks opportunity cost of the capital.

What if the resources could actually help shore up elementary, middle or high-school in low-income neighborhood? Or provide supplementary classes to those who are falling behind.

Failing to question the research, neglecting opportunity costs and blindly shoveling resources on moving a single metric will only result in moving the metric but with no tangible results.

Where do you stand?

Protected: Does Pinterest Really Generate $1.2 Billion in e-Commerce Sales?

This content is password protected. To view it please enter your password below:

Password:

When an idea stands alone, separated from its originator

Which one will have higher acceptance A) A random idea attributed to a popular Guru B) Popular Guru's idea attributed to a random person?

— Rags Srinivasan (@rags) February 9, 2012

I recently heard a NPR piece about the discovery of a copy of da vinci’s  Mona Lisa painting. Martin Bailey from The Art Newspaper had this to say in that piece

BAILEY: Well, I think maybe too much mystique has built up about this picture, the “Mona Lisa.” I mean, it is after all the world’s most famous painting. But people don’t look at it fresh. They look at it almost as an icon. And if you go to the Louvre, people aren’t actually really looking at the painting. They just want to sort of be in the same room with it. And for me, the beauty of the copy is that it actually makes us look at the painting as a painting, and I hope it will have that effect on other people too.

What Bailey says for world’s most famous painting and our reaction to it applies to famous ideas by famous gurus and our reactions to those.  We don’t seem to look at the ideas as just an idea when it is uttered by someone popular or with status.   As Galbraith wrote, anyone with position is assumed to be gifted with deep insights.

We may not grasp the idea nor we may analyze whether or not it is applicable to our case. But we just want to be part of the “conversation”. We tweet, retweet and write about it just to be in the same room as the Guru and his idea.

What if we are able to separate the idea from the one who said it?

What if I copied the idea of a Guru, word for word, intonation for intonation and stated it as mine? Will the mere act of copying make it stand alone?

Will we see the idea for what it is with all its biases?

 

Measure what is relevant, not what is available, convenient and sounds magical

Recently I saw a TV ad for a vitamin supplement. I do not know whether it is FDC approved or not. The Ad shows a woman drinking orange juice directly from the container (about 1 gallon). As she guzzles it with pain, the voice over says,

“You have to drink a whole gallon of orange juice to get the amount of vitamin-C in a single tablet of of our brand of Vitamin-C. It is the only tablet that gives you vitamin-C without the calories”

Very likely a true statement. But is it relevant? A single tablet of most such supplements has 1000mg of Vitamin-C and a single 8oz glass of orange juice  has 30-60mg. So a gallon bottle has 480-960mg of Vitamin-C. So a true claim.

What is our daily need for Vitamin-C? About 30mg. What will your body do with all those extra 970mg of Vitamin-C but to excrete it? Even if our body keeps the additional 970mg, what benefit comes of it? or worse, what are the side effects? What about other nutrients present in the glass of orange juice that are not accounted for?

This is  clever messaging, focusing customers attention on metrics that are readily available, prominent, convenient and advantageous to you even though the metrics may be completely irrelevant.
It is easy to dismiss such metrics when the claims are outrageous, when someone without authority/power/pedigree states them or after someone breaks it down. However, what if such a claim is stated by someone in authority, with pedigree/title or with some semblance of data and analysis?

1. Group Buying sites that tout their ability to deliver an year worth of traffic to businesses – But at what cost?
2. Marketing Managers who quote the number of times their product story was quoted in media and blogs – But at what cost and what is the incremental revenue?
3. A Social Media guru urging businesses to invest in social media presence and measure the investment’s sucess with a new metric “Return on Engagement” – But at what cost and what is the incremental revenue?
4. Shifting investor and market’s attention to growth in deferred revenue – but what if it came from pulling in future sales at a discount?
5. Measure how big the customer smile is after they leave your store, it is a clear indication of profitability (okay I made this up, but the rest are not)

Proponents of such spurious metric and their followers are likely to answer the cost and incremental revenue question with, “in the long term it will pay off”. Your questions should be, How long a long term?, What other options would deliver the same “long term return”?, What is the long-term return?
Do you measure what is relevant or what is prominent, convenient, available, magical and irrelevant?