This has been making the rounds in the neuroscience world, but deserves attention in cross-disciplinary fields. A group of top-notch fMRI researchers presented an unusual paper at June’s Human Brain Mapping conference.
Blog headline: fMRI Gets Slap in the Face with a Dead Fish
In short, researchers scanned a dead fish while it was “shown a series of photographs depicting human individuals in social situations. The salmon was asked to determine what emotion the individual in the photo must have been experiencing.”
Clearly, the fish did not perform well at the task, and thus we have not learned much about interspecies perspective taking. The work is, however, a compelling and humorous demonstration of the problem of multiple comparisons. This is a principle in statistics that basically says when you’re looking at enough bits of information (i.e. doing lots of statistical tests), some will seem to be what you’re looking for – purely by chance. In fMRI experiments, there are a LOT of pieces of data to compare, and without statistical correction for this phenomenon (which is not always done), some will indeed be significant, just by chance.
Lead author Craig Bennett explains further on his blog:
In early 2008 I was working with my co-adviser George Wolford on a presentation he was giving regarding the multiple comparisons problem in fMRI. We were discussing false positives in MRI phantom data and I brought up the idea of processing the salmon fMRI data to look for some ‘active’ voxels. I ran the fish data through my SPM processing pipelines and couldn’t believe what I saw. Sure, there were some false positives. Just about any volume with 65,000 voxels is going to have some false positives with uncorrected statistics. Rather, it was where the false positives occurred that really floored me. A cluster of three significant voxels were arranged together right along the midline of the salmon’s brain.
Remember that the fish was dead. There was surely no BOLD signal changes going on in a dead fish’s brain. This is likely not a physiological artifact; it is a statistical one. Furthermore, the voxels were clustered together – something that may be expected to happen in an “actual” activation and thus used as a threshold for analysis. Also, it was just one fish! (No apparent speculation in the paper about what may have happened if this were a school of fish compared to appropriate control school of fish.)
Bennett et al are apparently having a hard time getting the paper published. The use of multiple comparisons corrections in fMRI studies is a contentious one, as some researchers think it may be overly conservative and thus miss true positives. As a solution, Bennett suggests reporting both sets of data, corrected and uncorrected.
The moral of the story for interdisciplinary folks: note whether multiple comparisons correction data have been reported (or not). And always bear in mind that there are a lot of assumptions and decisions being made behind the ultimately reported data in any neuroimaging study.
– Emily Murphy (h/t Alexis Madrigal @ Wired)