[From the working paper “The Dozen Things Experimental Economists Should Do (More Of)” by Eszter Czibor, David Jimenez-Gomez, and John List, Working Paper 25451 in the NBER Working Paper Series]

“…We believe that the best approach to increasing the reliability of results from experimental economics lies in replication.”

“…Moonesinghe et al. (2007) and Maniadis et al. (2014) demonstrate how a few successful replications can increase the positive predictive value of a finding. This increase is particularly dramatic in cases when prior probabilities are low. Within the same framework, Coffman and Niederle (2015) argue that even the most inaccurate beliefs can be corrected within three to five replications.”

“…Replications typically bring little recognition for their authors despite the substantial work they entail. The process is particularly tedious because data and code for published articles are often unavailable – even though most leading economics journals have introduced data sharing requirements and mandatory data archives, such policies are not necessarily enforced (Hoeffler, 2017).”

“…studies suggest that the natural rate of replication in empirical economics is low, although heavily cited and influential papers do tend to get replicated.”

“… The first systematic evidence of replicability of laboratory experiments in economics is provided by Camerer et al. (2016) …They find a significant effect in the same direction as in the original study in 11 out of the 18 studies, corresponding to a reproducibility rate of 61%. …The authors also compare the replicated effect sizes with the original, and find a mean relative effect size of 65.9%.”

“Another noteworthy replication initiative is the Social Sciences Replication Project, … They find a significant effect in the same direction as the original study for 13 (62%) studies, and the effect size of the replications is on average about 50% of the original effect size (Camerer et al., 2018). Finally, while both of the above-mentioned projects focus on results published in top journals, Maniadis et al.(2017) analyze replication attempts from 150 economic journals, and find a “success rate” of 42.3% among the 85 experimental replication studies in their sample.”

To read the working paper, click here. (NOTE: This paper is behind a paywall.)