Replication is key to the credibility and confidence in research findings. As falsification checks of past evidence, replication efforts contribute in essential ways to the production of scientific knowledge. They allow us to assess which findings are robust, making science a self-correcting system, with major downstream effects on policy-making. Despite these benefits, reproducibility and replicability rates are surprisingly low, and direct replications rarely published. Addressing these challenges requires innovative approaches in how we conduct, reward, and communicate the outcomes of reproductions and replications.

That is why we are excited to announce the official launch of the Institute for Replication (I4R), an institute working to improve the credibility of science by systematically reproducing and replicating research findings in leading academic journals. Our team of collaborators supports researchers and aims to improve the credibility of science by

– Reproducing, conducting sensitivity analysis and replicating results of studies published in leading journals.

– Establishing an open access website to serve as a central repository containing the replications, responses by the original authors and documentation.

– Developing and providing access to educational material on replication and open science.

– Preparing standardized file structure and code and documentation aimed at facilitating reproducibility and replicability by the broader community.

How I4R works

Our primary goal is to promote and generate replications. Replications may be achieved using the same or different data and procedures/codes, and a variety of definitions are being used.

While I4R is not a journal, we are actively looking for replicators and have an ongoing list of studies we’re looking to be replicated. Once a set of studies has been selected by I4R, our team of collaborators will confirm that the codes and data provided by the selected studies are sufficient to reproduce their results. Once that has been established, our team recruits replicators to test the robustness of the main results of the selected studies.

For their replication, replicators may use the Social Science Reproduction Platform. We also developed a template for writing replications which is available here. This template provides examples of robustness checks and how to report the replication results. Once the replication is completed, we will be sending a copy to the original author(s) who will have the opportunity to provide an answer. Both the replication and answer from the original author(s) will be simultaneously released on our website and working paper series.

Replicators may decide to remain anonymous. The decision to remain anonymous can be made at any point during the process; initially, once completed or once the original author(s) provided an answer. See Conflict of Interest page for more details.

We will provide assistance for helping replicators publish their work. Replicators will also be invited to co-author a large meta-analysis paper which will combine the work of all replicators and answer questions such as which type of studies replicate and what characterizes results that replicate. For more on publishing replications, keep reading!

We need your help

I4R is open to all researchers interested in advancing the reproducibility and replicability of research. We need your help reproducing and replicating as many studies as possible. Please contact us if you are interested in helping out. We are also actively looking for researchers with large networks to serve on the editorial board, especially in the field of macroeconomics and international relations for political science.

Beyond helping out with replication efforts, you can help our community by bringing replication to your classroom. If you want to teach replication in class assignments, our team has developed some resources that might be of interest. A list of educational resources is available here.

A very useful resource is the Social Science Reproduction Platform (SSRP) which was developed by our collaborators at the Berkeley Initiative for Transparency in the Social Sciences in collaboration with the AEA Data Editor. This is a platform for systematically conducting and recording reproductions of published social science research. The SSRP can be easily incorporated as a module in applied social science courses at graduate and undergraduate levels. Students can use the platform and materials with little to no supervision, covering learning activities such as assessing and improving the reproducibility of published work and applying good coding and data management practices. Guidance for instructors such as how to select a paper, timelines and grading strategy is available here.

Reach out to us if you want to learn more about the SSRP and other teaching resources. We are here to help!

Where to publish replications

Incentives for replications are currently limited, with a small number of replications published in top journals. Moreover, reproducing or replicating others’ work can lead to disagreements with the original author(s) whose work is re-analyzed. One of I4R’s main objective is to address these challenges and help researchers conduct and disseminate reproductions and replications.

As a first step to better understand publication possibilities for replicators, our collaborators (Jörg Peters and Nathan Fiala) and the Chair, Abel Brodeur, have been contacting journal editors for top economic, finance and political journals asking them whether they are willing to publish comments for papers published in their journal and/or comments on studies published elsewhere. The answers are made publicly available on our website. We also highlight special issues/symposiums dedicated to replications and journals which strictly publish comments. Please contact us if you want to advertize other replication efforts or special issues related to open science and replications.

We will continue developing new and exciting features based on input from the community. Do not hesitate to reach out to us!

RESOURCES: https://i4replication.org/contact.html, Twitter @I4Replication

Abel Brodeur is Associate Professor in the Department of Economics, University of Ottawa, and founder and chair of the Institute for Replication (I4R). He can be reached at abrodeur@uottawa.ca.