The journal Economics: The Open Access, Open Assessment E-Journal is publishing a special issue on “The Practice of Replication.” This is how the journal describes it: 

“The last several years have seen increased interest in replications in economics.  This was highlighted by the most recent meetings of the American Economic Association, which included three sessions on replications (see here, here, and here). Interestingly, there is still no generally acceptable procedure for how to do a replication.  This is related to the fact that there is no standard for determining whether a replication study “confirms” or “disconfirms” an original study. This special issue is designed to highlight alternative approaches to doing replications, while also identifying core principles to follow when carrying out a replication.”

“Contributors to the special issue will each select an influential economics article that has not previously been replicated, with each contributor selecting a unique article.  Each paper will discuss how they would go about “replicating” their chosen article, and what criteria they would use to determine if the replication study “confirmed” or “disconfirmed” the original study.”

“Note that papers submitted to this special issue will not actually do a replication.  They will select a study that they think would be a good candidate for replication; and then they would discuss, in some detail, how they would carry out the replication.  In other words, they would lay out a replication plan.”

“Submitted papers will consist of four parts: (i) a general discussion of principles about how one should do a replication, (ii) an explanation of why the “candidate” paper was selected for replication, (iii) a replication plan that applies these principles to the “candidate” article, and (iv) a discussion of how to interpret the results of the replication (e.g., how does one know when the replication study “replicates” the original study).”

“The contributions to the special issue are intended to be short papers, approximately Economics Letters-length (though there would not be a length limit placed on the papers).” 

“The goal is to get a fairly large number of short papers providing different approaches on how to replicate.  These would be published by the journal at the same time, so as to maintain independence across papers and approaches.  Once the final set of articles are published, a summary document will be produced, the intent of which is to provide something of a set of guidelines for future replication studies.”

Despite all the attention that economics, and other disciplines, have devoted to research transparency, data sharing, open science, reproducibility, and the like, much remains to be done on best practice guidelines for doing replications.  Further, there is much confusion about how one should interpret the results from replications.  Perhaps this is not surprising.  There is still much controversy about how to interpret tests of hypotheses!  At the very least, it is helpful to have a better understanding of the current state of replication practice, and how replicators understand their own research.  It is hoped that this special issue will help to progress our understanding on these subjects.

To read more about the special issue, and how to contribute, click here.