Software Flaws May Invalidate 40,000 Neuroscience Studies

The human fallibility of science has been in the news a lot lately. The latest hit? Researchers say the three most popular programs for interpreting functional magnetic resonance imaging (fMRI) results all produced shockingly high rates of false positives—a finding that could invalidate tens of thousands of studies. The report by researchers from Sweden's Linköping University and the UK's University of Warwick was published in the Proceedings of the National Academy of Sciences.

Over the last two and a half decades, fMRI has become a crucial tool for investigating the brain and its inner workings. Because our bodies supply blood to the parts that need it most, fMRI’s ability to measure blood flow in the brain allows us to track brain activity. But just how good of a job is it doing? Scientists weren’t really sure.

“Functional MRI (fMRI) is 25 years old, yet surprisingly its most common statistical methods have not been validated using real data,” the authors write.

To test it, they pulled data on 499 healthy people from the 1000 Functional Connectomes Project, a database of baseline brain scans. They then split their subjects into different subgroups and compared the scans in different permutations, over and over, until they had 2,880,000 different group analyses.

For a long time, neuroscientists have believed that scanning the brain of a healthy person in a resting state should yield no signs of activity. Consequently, any sign of activity could be read as a positive result.

But the authors of the current study say that assumption could be wrong, and following it could lead to a lot of false positives.

Looking at the results of their almost 3 million comparisons, the researchers expected to find a false positive rate of about 5 percent. Instead, they found that the top three fMRI analysis programs—Statistical Parametric Mapping, the fMRIB Software Library, and Analysis of Functional Neuroimages—yielded false positive rates up to 70 percent. That’s right: seven out of 10 scans analyzed using these methods came to inaccurate conclusions. And the authors say that may be a conservative estimate.

If these findings are correct, it has a huge impact on neuroscience research. In a literature search, the authors found more than 40,000 neuroscience studies that relied on fMRI results. Still, there may be no point in going back and re-doing these studies. It’s better to look to the future, the authors say: "Considering it is now possible to evaluate common statistical methods using real fMRI data, the fMRI community should, in our opinion, focus on validation of existing methods.”

Know of something you think we should cover? Email us at tips@mentalfloss.com.