Recently I read a blog post by the Buffer team. The post is about getting better click through rates from links included in tweets. They titled it as scientific and later refer to it as scientifically proven method. I do not understand why this is scientific, what the findings really mean or what are the real implications.
For one thing scientific method is distinguished by its search for falsifiable evidence. Scientific rigor requires you not to stop when you find something interesting or fits your preconceived notion but look for data, if found, that will falsify the original notion. For instance biologists would say the falsifiable evidence for theory of natural selection is rabbit fossil found in Precambrian period.
Even if you set aside the request for such rigor from a blog post looking for inbound traffic you would at least expect them to follow what fourth graders learn as scientific method – make a hypothesis, run an experiment, collect data,analyze … ( let us not forget that we cannot stop when data fits the hypothesis data can fit any number of hypotheses. But we will once again forgive a blog post looking for links and retweets)
What I see in their recent post on CTR for tweets is, they analyzed gobs of data, found some interesting differences and called it as theory of better tweets. I have written about the perils of data dredging and how large samples magnify even small differences.
I think they have within their means to make their method scientific, or more scientific than it is now. I would like to make following request to the buffer team, in the name of science
- One of the pillars of scientific method is verifiably. Please make the data set available to others. Not all of it. Pick just 300 random samples and share.
- since no one I had asked before ever shared data on their social media science I will make the next simpler request.
For all the tweets you collected find the base rate for CTR and the standard deviation.
For the category you define as Adverb, pick 300 random samples, find the sample mean and S.D.
Same for the category you define as Adjective.
Do a t-test between the two samples to see if the differences are indeed statistically significant.
If indeed they are, then you can say a scientific method.
- here is even simpler request. Calculate the base rate for CTR. Then tell us the posterior probability of click-through if I used adverb or adjective.
Bear in mind though, statistical significance does not mean economic significance. What is your opportunity cost of spending time cramming your tweets with a verbs? And what would happen when everyone follows your advice and use all adverbs?