Fitbit, the maker of bands that count steps, recently made following claims from its two year “study”,
after two years, employees who opted in to the wearable program cost on aver- age $1,292 less than employees in the control group.
Sounds plausible? Or even compelling for businesses to make their employees wear $150 Fitbit devices for reaping close to 10X healthcare savings?
Not so fast. let us look closely at the reported results and more importantly the so called study.
- Single clustered sample – The measurements are done on employees of a single corporation. Regardless of the size of the test corporation, this is not random sampling as geography, demographics and several other factors adversely affect the results. Any savings claimed from the study are not applicable generically to other companies.
- Opt-in Study – The study asked for volunteers to opt-in to wear subdisdized Fitbit bands. Those who opted-in were called “treatment” group and those didn’t are called the “control” group. This is simply the wrong method for measuring effect of a “treatment” because it is like putting your thumb on the scales to favor the treatment group.
- Not controlling for other variables – For results of a study to be attributable to specific treatment, the researcher must control for all other variables. That is not the case with this study. For starters the people who opted-in may already be on their path to better health, or may be doing a few other things like eating right and regular vigorous exercise. On the flip side, those who did not opt-in to wear the Fitbit may have other illness that prevent them from walking 10000 steps. That is, there are too many confounding variables that Fitbit did not control for in this study.
- Where is the placebo?- If they are studying the effect of their Fitbit band the control group must be given a placebo like another band or dummy band that fakes step counting. That is how done in correctly executed randomzied control trials. You can see the self-serving nature of this study. Had they used the $18 band like Xiaomi as placebo or even a simple bracelet that fakes step counting they would have found no statistically significant advantage to using Fitbit.
- Using really large samples – You may be surprised to see this as a flaw in the study as most often you hear too small sample size as the issue. The so called treatment group had 905 people and the control group had 1784 people. As I have explained before, samples larger than 300 exaggerate differenes that are not there. Had they randomly picked 300 people from these two groups and compared results, that would be more defensible than this large sample comparison.
The net is, the results are meaningless and not actionable because the study is flawed.
Finally I leave you with this wisdom from authors of a real clinical study on these wearable step-counters,
“The notion you can give out a bunch of watches and suddenly people will get more active is just silly.”
Or they would get healthy and save $1300 a year is downright outrageous.