10 Problems with Foursquare Big Data Prediction of iPhone 6s Sales

Foursquare, an App that tracks your movements and checkins, made a prediction on launch weekend iPhone 6s sales. It is a good PR to latch on to a major event to showcase their new entry into data analysis. This is also going to be used as an example of Big Data analysis. Since Foursquare has access to time series data on locations of its users, it is argued that they can put that data to work to “unlock value from it”.  Combine this movement with location of stores, you have foot traffic information on stores which they believe is a powerful dataset.

Here is how Foursquare made the prediction,

So our Place Insights team looked into foot traffic at Apple stores leading up to the launch of the iPhone 5, 5S, and 6 and analyzed it alongside Apple’s public sales data.

Combining Foursquare’s foot traffic data with Apple’s sales data on a graph shows how closely the two are linked. Visit growth is clearly a strong general advance indicator for sales performance of the launch weekend.

Foursquare predicts that launch day foot traffic will be about 360% of a typical Friday. This likely means that Apple will sell 13–15 million iPhones this weekend,

But is the method valid? Is the prediction trustable? The answer is NO to both questions. This is yet another case of chasing Big Data and letting spurious correlations make seemingly correct predictions. Here are the problems with Foursquare’s method and prediction:

1. Does foot traffic mean sales? Isn’t it possible more simply visit the store during launch weekend to look at the new phones?
2. What percentage of people who walk into Apple stores have Foursquare app? Is that a representative sample.
3. What other factors will drive foot traffic? This time there is iPad Pro which is going to bring in more people who may want to see it in action.
4. Problem with iPhone 5s data used by Foursquare. iPhone 5s was not offered for preorder and did not put up big numbers in its initial quarter.
5. What percentage of iPhone sales happen in Apple stores? Even if we accept point 1 above we should scope the effect of Apple stores on total sales. According to a 2012 survey,  Apple Stores accounted for just 21% of the iPhone Sales. Even if this is different for opening weekend that explains only portion of sales.
6. More sales are shifting online. Apple said the online demand for iPhone 6s Plus has been exceptionally strong and exceeded their own forecasts for the preorder period.
7. More sales happening though other channels. T-Mobile CEO stated iPhone 6s pre-orders on T-Mobile are up an impressive 30% compared to iPhone 6 pre-orders last year.
8. Changes in pricing model affect sales. This year subscription pricing for iPhone 6s is available from all carriers and even from Apple. There is a price war on monthly subscription fee between T-Mobile and Sprint. These factors drive new customers that is going to increase launch weekend volume.
9. China. China China. Last two times Apple did not launch in China at the same time. iPhone 6s was available in China at the same time as US.
10. Finally there is no information on confidence level or likelihood information in the prediction. I give them credit for predicting a range vs. one absolute number. But they do not state at what confidence level. That is the difference between big data analysis vs. statistical analysis.

What we have here is a faulty method that falls apart despite Foursquare’s access to largest database of information on the foot traffic of people around the globe. All that big data does not help them see Madagascar from San Diego.

Could the rest of us make better predictions without the Big Data? Yes we can if we do scenario analysis with the new factors that affect iPhone 6s sales. And we can state not only a range but also at what confidence level.

Big Data has applications but this is not it. Foursquare has bigger ambitions to monetize all their big data by making similar predictions for Marketing, Real Estate, Finance and Credit Scoring. All these predictions will suffer from the same challenges unless you build a more comprehensive model with foot traffic data as just one part of prediction. Otherwise we will be looking for Shamu in Madagascar.

That is some messed up data

This is a guest post by Robert Moss. Robert is a freelance copywriter who writes traditional and non-traditional advertising for both consumer and business to business brands. Robert has an impressive portfolio and customers. Robert tweets at @Moss_Robert. Robert suggested a  more colorful title to match his topic but I had to use “messed” instead of his suggestion.

Have you written your guest post yet? Time is running out.

---

According to Marchex, a mobile advertising company, people in Ohio swear more than any other state. Marchex analyzed data from more than 600,000 phone calls placed to customer service centers and businesses in 30 industries. The calls were to cable and satellite companies, auto dealerships, pest control centers and others.

Who doesn’t hate their cable company? In addition to Comcast and Time Warner, Ohio appears to have at least 12 more local and regional cable companies to hate in addition to the major cable companies. Auto dealerships are local, and pest control centers can be national, local or anything in between.

So does Marchex’s study reveal which state has the most potty mouths? Or does it reveal which state has the worst customer service when it comes to cable companies, auto dealerships, pest control centers and other companies? Who knows? There may be other factors that make life miserable for Ohioans and cursing may be a symptom of that misery and not an end in itself. Poor nutrition, bed bugs and one of the worst states to grow old may explain Ohio’s high level of cursing. Of course while correlation may not equal causation, Marchex’s study provides more questions than answers.

What all this does seem to reveal is that Marchex came up with a study that generates lots of press about their company. If that was what they intended to measure, then they succeeded.

Today we use reasons, Or do we?

What is a predictor for a political candidate’s success? It is in the millions of tweets about the contest.

What will get more customers to buy? It is in getting them to like your brand’s facebook page.

How can you get your message across? It is by getting all your employees tweeting in unison.

What is a predictor of getting your tweet retweeted? The secret is in using more adverbs (or is it adjectives)

When Stephen Hawking wrote about our early models about the universe he wrote,

Ignorance of nature’s ways led people in ancient times to postulate many myths in an effort to make sense of their world. According to Viking mythology, eclipses occur when two wolves, Skoll and Hati, catch the sun or moon. At the onset of an eclipse people would make lots of noise, hoping to scare the wolves away.

If you look at it Bigdata of these days is no different from what those Vikings in early days did trying to understand eclipse. Bigdata or  not is defined not by volume (ok velocity, variety) but by our ability to process with tools at our disposal and how advanced our mind and reasoning is.  It is not farfetched to say,  our early ancestors when bombarded with cosmic data saw that as Bigdata problem, trying to make some sense out of it with myths.

Hawking was very optimistic and forgiving by adding that

Today we use reason, mathematics and experimental test—in other words, modern science.

Today we have no scarcity of data, we have abundance of it – billions of tweets, facebook updates, instagram pictures – all volume, variety and velocity. Do we really use reason,  (right )mathematics and experimental test to make sense of this social media noise?

May be Hawking is true in case of science but not in social media science, data science or in so called science of marketing. If not would be seeing any of those statements we saw in the beginning of this article, propagated as “scientifically proven” methods and retweeted by millions?

 

 

Big Data predicts people who are promoted often quit anyway – But …

I saw this study from HP that used analytics (okay Big Data analytics, whatever that means here) to predict employee attrition.

HP data scientists believed a companywide implementation of the system could deliver $300 million in potential savings “related to attrition replacement and productivity,

I must say that unlike most data dredging that goes on with selective reporting these data scientists started with a clear goal in mind and a decision to change before diving into data analysis. It is not the usual

“Storage and compute are cheap. Why throw away any data? Why make a decision of what is important and what is not? Why settle for sampling when you can analyze them all? Let us throw in Hadoop and we will find something interesting”

Their work found,

Those employees who had been promoted more times were more likely to quit, unless a more significant pay hike had gone along with the promotion

The problem? Well this is not the hypothesis they developed independent of data and then collected data to test this. That is the prescribed approach to hypothesis driven data analysis. Even with that method one cannot stop when data fits the hypothesis because data can fit any number of plausible hypotheses.

The problem is magnified with Big Data where even tiny correlations get reported because of sheer volume of data.

What does it mean that people who are promoted often quit?

Is it the frequent promotion that is the culprit? Isn’t it likely that those who are driven and high value-add more likely to get promoted often,  likely to want to take on new challenges and also look attractive to other companies?

The study adds, “unless associated with a more significant pay hike”.

Isn’t it more likely that either the company is simply using titles to keep disgruntled employees happy or just making up titles to hold on to high performance employees without really paying them for the value add? In either case, aren’t the employees more like to leave after few namesake promotions that really don’t mean anything?

Let us look at the flip side. Why are people who are not promoted frequently end up staying?  Why do companies give big raises to keep those who were promoted?

Will stopping frequent promotion stop the attrition? Or will frequent promotion with big pay raises stop it? Neither will have an effect.

The study and the analysis fail to ask
Is the business better off paying big raises to keep those who are frequently promoted than letting them leave?

Is the business better off if those who are not promoted often choose to stay?

That is the problem with this study and with Big Data analytics that do not start with a hypothesis developed outside of the very data they use to prove it. It finds the first interesting correlation, “frequent promotions associated with attrition” and declares predictability without getting into alternate explanations and root cause.

Big Data does not mean suspension of mind and eradication of theory. The right flow remains

1. What is the decision you are trying to change?
2. What are the hypotheses about drivers- developed by application of mind and prior knowledge?
3. What is the right data to test?
4. When data fits hypothesis could there be alternate hypotheses that could explain the data?
5. How does the hypotheses change when new data comes in?

How do you put Big Data to work?

My First GitHub Commit – Select Random Tweets

I have seen several reports that collect and analyze millions of tweets and tease out a finding from it. The problem with these reports is they do not start with any hypothesis and find the very hypothesis they are claiming to be true by looking at large volumes of data. Just because we have  Big Data, it does not mean we can suspend application of mind.

In the case of twitter analysis, only those with API skills had access to data. And they applied well their API skills to collect every possible tweet to make their prediction that are nothing more than statistical anomalies. Given millions of tweets, anything that looks interesting will catch the eye of a determined programmer seeking to sensationalize his findings.

I believe in random sampling. I believe in reasonable sample sizes, not whale of a sample size. I believe that abundance of data does not obviate the need for theory or mind. I am not alone here. I posit that the any relevant and actionable insight can only come from starting with relevant hypothesis based on prior knowledge and then using random sampling to test the hypothesis. You do not need millions of tweets for hypothesis testing!

To make it easier for those seeking twitter data to test their hypotheses I am making available a real simply script to select tweets at random. You can find the code at GitHub. You can easily change it to do any type of randomization and search queries. For instance you want to select random tweets that mention Justin Bieber, you can do that.

The script has bugs? I likely does. Surely others can pitch in to fix it.

Relevance not Abundance!

Small samples and test for statistical significance than all the data and statistical anomalies.