The problem of bias in science

 
December 12, 2017

Here's a Scientific American article ("Bad News for the Highly Intelligent," which reports on a study claiming a correlation between high IQ and mental and physical disorders) that, indirectly, serves as a pretty good illustration of the problem of bias in science.

Why? Well let's take a look at *method* of the study being reported upon:

The researchers have drawn their "high-intelligence" study group from the ranks of Mensa, which defines itself as having a membership consisting of folks who score in the top two percentiles of the population on certain standardized intelligence tests.

It certainly sounds like a reasonably convenient way to reach a large number of people who, by some measure, can be classified as "highly intelligent." But the word convenient there should be a red flag: one of the Introduction-to-Statistics terms that most scientists (should) learn is that of "convenience sample". In short, a sample which is convenient is not necessarily representative.

How might that apply here? Well there's the hidden bias: Mensa may define itself as "members who are very intelligent by a certain measure," but the unstated criterion is "and who want to be in Mensa." That last bit may sound obvious, but it's actually IMPORTANT. Let's look at some basic numbers:

1) The US has a population of around 320 million people. By definition, this means that around 6.4 million people are the "top 2%" in anything, in this case, "intelligence as measured by standardized tests."

2) Mensa in the US has "over 50,000 members." As long as the number is less than 64,000, this is less than 1% of people who could presumably qualify.

3) If 2% of the population qualifies, but only 1% of those qualifiers join, then it is arguable that the desire to join Mensa is actually more of a determining factor of who's in Mensa than the intelligence-test requirement.

This last point is doubly true when one recognizes that intelligence tests can be, frankly, learned: someone who wants to improve their results can do so with practice - i.e. someone who feels compelled to get a higher score is more likely to get a higher score than someone with no particular urge to make a score of more than (say) 132.

To the credit of the Scientific American article, this factor is... eventually... mentioned, well down in the text: "It’s also possible that people who join Mensa differ from other people in ways other than just IQ" (though it goes to talk about an "intellectual lifestyle" rather than "the urge to be in a selective-on-intelligence-test society"). Oddly, this seems to be more acknowledgment of the potential bias than the scientific paper itself, which gives a defense focused upon how much more difficult it would be to obtain a high-intelligence-test-score sample without resorting to an already-existing group (which is true, and also practically an exemplar of "convenience sample").

Note: kudos to the authors of the scientific study "High intelligence: A risk factor for psychological and physiological overexcitabilities" for publishing in an open-access format, which makes a comparison of what the study says and what the article about the study says possible.

The point, in summary, is this: the base scientific study, and the Scientific American article, lead with
"high IQ is correlated with mental and physical disorders"

when in fact it would be just as accurate, if not more so, to say
"desire to be in an exclusive high-IQ organization is correlated with mental and physical disorders."

and acknowledging this interpretation seems a prerequisite for looking for any causative factors that may be illuminated by such a correlation.  (Famously, of course, "correlation does not equal causation," but it's where scientists often look for clues to causes.)