[Excerpts taken from the article “Variability in the analysis of a single neuroimaging dataset by many teams” by Thomas Nichols, Russell Poldrack, Tom Schonberg and 194 others, posted at BioRXiv]

“Data analysis workflows in many areas of science have become exceedingly complex, with a large number of processing and analysis steps that involve many possible choices at each of those steps (i.e., “researcher’s degrees of freedom”).”

“Simulation studies have shown that these differences in analytic choices can have substantial effects on results, but it has not been clear to what degree such variability exists and how it affects reported scientific conclusions in practice. Recent work in psychology has attempted to address this through a “many analysts” approach, in which the same dataset was analyzed by a large number of groups…”

“In the Neuroimaging Analysis Replication and Prediction Study (NARPS; http://www.narps.info), we applied a similar approach to the domain of functional magnetic resonance imaging (fMRI), where analysis workflows are complex and highly variable.”

“Neuroimaging researchers were solicited via social media and at the 2018 annual meeting of The Society for Neuroeconomics to participate in the analysis of this dataset. Seventy analysis teams participated in the study.”

“The teams were provided with the raw data…as well as optional preprocessed data…They were asked to analyze the data to test nine ex-ante hypotheses (Table 1), each of which consisted of a description of significant activity in a specific brain region in relation to a particular feature of the experimental design.”

“They were given up to 100 days (varying based on the date they joined) to analyze the data and report for each hypothesis whether it was supported based on a whole-brain corrected analysis (yes / no)….the only instructions given to the teams were to perform the analysis as they usually would in their own research groups and to report the binary decision based on their own criteria…”

“The fraction of teams reporting a significant result for each hypothesis is presented in Table 1…. The extent of the variation in results across teams can be measured as the fraction of teams reporting a different result than the majority of teams.”

“The second aim of NARPS was to test whether peers in the field could predict the results obtained in aggregate by the analysis teams using prediction markets.”

“…we ran two separate prediction markets: one involving members from the analysis teams (“team members” prediction market) and an additional independent market for researchers in the field who had not participated in the analysis (“non-team members” prediction market).”

“Overall, n = 65 traders actively traded in the “non-team members” prediction market and n = 83 traders actively traded in the “team members” prediction market.”

“Except for the prediction of a single hypothesis (Hypothesis #7) in the “team members” set of markets, all predictions were outside the 95% confidence intervals of the fundamental values (i.e. the proportion of teams reporting a significant result for each hypothesis; see Figure 3…”

Discussion

“The analysis of a single functional neuroimaging dataset by 70 independent analysis teams revealed substantial variability in reported binary results, with high levels of disagreement across teams of their outcomes on a majority of tested pre-defined hypotheses.”

“Prediction markets demonstrated that researchers generally overestimated the likelihood of significant results across hypotheses, even those who had analyzed the data themselves, reflecting substantial optimism bias by researchers in the field.”

“Given the substantial amount of analytic variability we found to be present in practice, leading to substantial variability of reported hypothesis results with the same data, we believe that steps need to be taken to improve the reproducibility of data analysis outcomes.”

To read the article, click here.