[H/T to Retraction Watch for the material in this post] This past week, the journal Science published a study by Dan Gilbert, Gary King and others refuting the claims of Brian Nosek and the Reproducibility Project Psychology (RPP). The RPP study replicated 100 experiments in psychology and found that over half could not be replicated. Gilbert, King and associates reviewed the evidence behind that study and found it did not support the RPP’s pessimistic conclusions. But that is only the beginning of the story. In the days following, there have been numerous internet articles examining the two studies, trying to determine whether there is a replicability crisis in psychology or not. It turns out that things are not so clear cut, though it appears the original claims of the RPP overstated the replication failure rate. The following articles are a good sampling of this debate: A review of the studies is given here. An article that features a response by Nosek of the RPP is given here. An article that puts the replicability debate into a larger context of a methodological/epistemogological struggle within Psychology is given here. And an article that argues everybody is missing the point is given here. Happy reading!
[FROM THE ARTICLE “STUDY THAT UNDERCUT PSYCH RESEARCH GOT IT WRONG” IN THE HARVARD GAZETTE] “In an attempt to determine the “replicability” of psychological science, a consortium of 270 scientists known as the Open Science Collaboration (OSC) tried to reproduce the results of 100 published studies. More than half of them failed, creating sensational headlines worldwide about the “replication crisis” in psychology. But an in-depth examination of the data by Daniel Gilbert, the Edgar Pierce Professor of Psychology at Harvard, Gary King, the Albert J. Weatherhead III University Professor at Harvard, Stephen Pettigrew, a doctoral student in the Department of Government at Harvard, and Timothy Wilson, the Sherrell J. Aston Professor of Psychology at the University of Virginia, has revealed that the OSC made some serious mistakes that make its pessimistic conclusion completely unwarranted.” To read more, click here.
[From the article “Evaluating replicability of laboratory experiments in economics” published in Science] Researchers today reported the results of a collaborative project in which 18 experimental economics studies originally published in the American Economic Review and the Quarterly Journal of Economics were replicated. The researchers report, “We find a significant effect in the same direction as the original study for 11 replications (61%); on average, the replicated effect size is 66% of the original.” This study compares with the recent Reproducibility Project Psychology (RPP) in which 100 psychology experiments were replicated. To read the article, click here. To read a summary of the article with commentary, click here.
In a recent article in Nature, DALMEET SINGH CHAWLA asks “How Many Replication Studies Are Enough?” The article highlights the comments of psychologist Courtenay Norbury whose work on autism in children has apparently been replicated numerous times. So much so that another researcher tweeted “It’s the opposite of the reproducibility crisis!” TRN says, “Must be a nice problem to have!” To read more from the article, click here.
Project TIER has announced the second round of its annual TIER Faculty Fellowships for leaders in the development and dissemination of curriculum related to documentation and replicability of social science research. Fellows cooperate with Project TIER in activities such as conducting workshops, organizing sessions at professional meetings, writing new print and electronic instructional materials, and promoting innovative methods of teaching concepts and methods of reproducible research. Each Fellowship carries a stipend of $5,000.00. Applications for AY 2016-17 Fellowships are due April 18. For more details, click here.
The journal International Studies Perspective has a collection of articles that address various dimensions of replication in the social sciences, in general, and the field of international relations, in particular. An interesting proposal comes from an article by MICHAEL COLARESI, who argues for “prereplication”: after an article is conditionally accepted, the authors are required to submit their data and code, and the journal confirms that the data and code are able to exactly replicate the results in the paper. This both ensures that the results in the paper are “locally correct,” and makes it easier for researchers to replicate the paper after it has been published. To read a summary of the different articles, click here.
A new paper by GARRET CHRISTENSEN, JUSTIN MCCRAY and DANIELE FANELLI in PLOS ONE suggests an alternative to using conventional t-values when researchers are concerned about publication bias. From the Abstract: “Publication bias leads consumers of research to observe a selected sample of statistical estimates calculated by producers of research. We calculate critical values for statistical significance that could help to adjust after the fact for the distortions created by this selection effect, assuming that the only source of publication bias is file drawer bias. These adjusted critical values are easy to calculate and differ from unadjusted critical values by approximately 50%—rather than rejecting a null hypothesis when the t-ratio exceeds 2, the analysis suggests rejecting a null hypothesis when the t-ratio exceeds 3.” To read more, click here.
(FROM THE ARTICLE “Psychology’s Replication Crisis Has a Silver Lining”) The author, a prominent professor of psychology at Yale University, argues that not all is gloom and doom. Some failures to replicate are due to noise swamping signal. Other failures to replicate are due to faulty replicators, not faulty original research. And much research IS replicable. Finally, as bad as things are in psychology, at least the situation is better than in medical research. To read more, click here.
[From the website The Edge] The question is: “What scientific concept would improve everybody’s cognitive toolkit?” Stanford Associate Professor BRIAN KNUTSON’s answer is: “Replication.” He suggests that a more meaningful measure of positive scientific impact is a Replication Index, or “r-factor.” This would measure the extent to which a researcher’s results could be independently confirmed. To boost their “r-factors”, researchers would be incentivized to make their work more transparent, making it easier for others to confirm their results. To read more, click here.
Replications are hot. Don’t believe me, just read The Economist. This is the third story related to scientific reliability/replications this past month. To read more, click here.
The Replication Network 
You must be logged in to post a comment.