You’re stuck in an elevator with someone who read a tweet about a study on women in boards

Let me start by repeating what I wrote a while back on the faulty analysis by Credit-Suisse,

Here is an undeniable fact – considering ~50% of the population is women and 50% of the board of directors are not women, we can safely say women are underrepresented in corporate boards. Now let us return to the reported study in question that makes a faulty case for adding women board members. My arguments are only with errors in the research methods and its application of faulty logic. Nothing more.

I will add to this my case against those who keep quoting such studies as evidence for their side.

The title of this article is inspired by Adrienne LaFrance. She wrote a blog post titled , “You’re stuck on an elevator with someone who loved that Sarah Lacy article. Now what?”

In that article Ms. LaFrance makes point by point argument to what seem to be silly questions from a clueless elevator companion who fell in love with Ms.Lacy’s post.  For one such question Ms.LaFrance quotes as evidence the research by Thomson-Reuters,

You should check out this study from earlier this year that showed how diverse corporate boards outperform those with no women. You’d think that a company like Twitter would put its business interests first.

She isn’t alone in quoting this study, almost everyone taking   Mr. Vivek Wadhwa‘s side use this study. I am not sure how many read this report or looked at its methods and caveats. Let me do that in this article.

Here is the link to the said research report.

1. Does the board matter?: The study starts with unverified assumption that a company’s board matters to its performance and then goes on to see differences in performance between boards. If your hidden hypothesis you took for granted is false it does not matter what your stated hypothesis is.
What the study does is, If A=TRUE,  then A(With Women) > A(Without Women)
You can see that if A =FALSE, the rest does not matter.You might want to stop here as nothing else matters after this error.
2. Control Variable: When you want to study the influence of a single variable you want to make sure all other variables are held constant. But when you read this report it is clear that they have no way to do that. They started with composition of a company board in 2007 then compared the performance of a group of companies over a period. There are two many uncontrolled variables during this period  (tech trends, market trends, industry verticals, etc.) and these affected different companies differently.
3. Error in Comparing Averages?: The comparisons are done on averages. There is a group of companies with mixed board and then there is another without women in board. The two groups are compared against another group, the benchmark which consist of companies of both kinds.

The report says companies with women on board did marginally better or same as the benchmark while those with no women on board did 10-15% lower than the benchmark.

First you notice that the difference in performance is not as significant s those who quote the study. Next you want to ask a simple clarifying question here.  If the benchmark has both types of companies, if one subset is  underperforming by 10-15%, shouldn’t the other subset outperform by 10-15% to bring it back to benchmark average?

The only explanation I can think of is average hides details here. There must be a few companies in each side that are significantly different from the arithmetic mean for that group and they account for the difference. If you leave out these samples and compare again, the difference will likely vanish.

Now to the question of what to do when stuck in an elevator with someone who merely heard about the Thomson Reuter’s study?

Just smile and nod.