In September of this year, the journal Economics: The Open Access, Open Assessment E-Journal published a series of Discussion Papers for a special issue on “The Practice of Replication”. The motivation behind the special issue came from the following two facts: First, there has been increasing interest in replications in economics. Second, there is still no standard for how to do a replication, nor for determining whether a replication study “confirms” or “disconfirms” an original study.

Contributors to the special issue were each asked to select an influential economics article that had not previously been replicated. They were to discuss how they would go about “replicating” their chosen article, and what criteria they would use to determine if the replication study “confirmed” or “disconfirmed” the original study. They were not to do an actual replication, but rather present a replication plan.

Papers were to consist of four parts: (i) a general discussion of principles about how one should do a replication, (ii) an explanation of why the “candidate” paper was selected for replication, (iii) a replication plan that applies these principles to the “candidate” article, and (iv) a discussion of how to interpret the results of the replication (e.g., how does one know when the replication study successfully “replicates” the original study). The contributions to the special issue were intended to be short papers, approximately Economics Letters-length (though there would not be a length limit placed on the papers). 

A total of ten papers were submitted to the special issue and have now been published online as Discussion Papers: nine replication plans and a general thought piece on how to do a replication. They are, respectively:

– Richard G. Anderson, “Should you choose to do so… A replication paradigm”

– B. D. McCullough, “Quis custodiet ipsos custodes?: Despite evidence to the contrary, the American Economic Review concluded that all was well with its archive”

– Tom Coupé, “Replicating ‘Predicting the present with Google trends’ by Hyunyoung Choi and Hal Varian (The Economic Record, 2012)”

– Raymond Hubbard, “A proposal for replicating Evanschitzky, Baumgarth, Hubbard, and Armstrong’s ‘Replication research’s disturbing trend’ (Journal of Business Research, 2007)”

– Andrew C. Chang, “A replication recipe: list your ingredients before you start cooking”

– Dorian Owen, “Replication to assess statistical adequacy”

– Randall J. Hannum, “A replication plan for ‘Does social media reduce corruption?’ (Information Economics and Policy, 2017)”

– Benjamin Douglas Kuflick Wood and Maria Vasquez, “Microplots and food security: encouraging replication studies of policy relevant research”

– Gerald Eric Daniels Jr. and Venoo Kakar, “Normalized CES supply-side system approach: how to replicate Klump, McAdam, and Willman (Review of Economics and Statistics, 2007)”

– Annette N. Brown and Benjamin Douglas Kuflick Wood, “Which tests not witch hunts: a diagnostic approach for conducting replication research”

The papers have been sent out for review and the reviews, along with the authors’ responses, are now beginning to appear online with the papers (the journal is open assessment). Before a decision is made on whether to publish the papers as articles, the journal would like to receive further comments from researchers interested in replications.

Online contributors can comment narrowly on whether a paper successfully carried out its fourfold task (see above). But they can also comment more generally on how they think a replication should be done, and/or how one should interpret the results from a replication.

The ultimate goal is to combine the different papers, the reviewers’ assessments, and the comments from online contributors to develop a set of guidelines for doing and interpreting replications. It is hoped that this “crowd-sourced” approach will bring a large range of perspectives to “the practice of replications.”

To contribute a comment on one or more of the papers, click on the papers’ links above and add a comment, which may require that you first register with the journal. Alternatively, you can email your comment to the special issue’s editor, W. Robert Reed, at bob.reed@canterbury.ac.nz. The deadline to submit comments is January 10, 2017.

Bob Reed is a professor of economics at the University of Canterbury in New Zealand and co-organizer of The Replication Network. He can be contacted at the email listed above.