What’s the International Initiative for Impact Evaluation (3ie) doing in the replication business? 3ie is mostly known in the development community as a funder of impact evaluations and systematic reviews. But our leadership always envisioned a role for replication research within 3ie’s mandate to provide high quality evidence for policymaking. We designed 3ie’s replication programme to encourage internal replication studies of influential, innovative, or controversial development-related impact evaluations. Through two rounds of replication windows we’ve funded 10 replication studies to date, including the highly discussed replication research around deworming treatments in Kenya (for a related news item in TRN, click here). So here’s what 3ie is doing in the replication business.

Our processes are designed to address common criticisms of replication research (with this blog borrowing from a forthcoming paper we’re writing about it). Selection of replication-eligible studies is fraught with insinuations of improper selection, with replication researchers supposedly only choosing to replicate studies they feel confident they can disprove. We address that criticism by creating different eligibility mechanisms, such as choosing studies based on a crowdsourced Candidate Studies List and gathering a committee of experts to judge the policy relevance of each replication study proposal.

One of the biggest concerns regarding replication research is researcher incentives. If bias exists for original authors to discover a new result, it can also exist for replication researchers to disprove the established result. To address this concern, we require all replication researchers to post replication plans, which allow readers to know how the researchers intended to undertake their replication study before starting the research. Ideally all robustness tests conducted in the replication paper will be publicly pre-specified in these replication plans.

In an attempt to further defuse replication tensions, we encourage engagement between the replication researchers and the original authors. We require 3ie-funded replication researchers to include a “pure replication,” and to share these results early in the replication process. In the pure replication, the researchers attempt to reproduce the published results using the same data and methods as in the publication. We then require these replication researchers to share their findings with the original authors before completing their study, giving the original authors the opportunity to reply to the direct reproduction of their work before any results are finalized.

Original authors are understandably sensitive to replication researchers who (in their opinion) solely aim to discredit their work. 3ie’s replication process includes multiple rounds of internal and external referring, including reviews of replication plans, pure replications, and draft final replication reports (see our peer reviewing replication research blog for more detail).

Finally, replication researchers are concerned that they will spend a significant amount of time conducting their study and then have no place to publish it. And original authors are worried that they won’t have an opportunity to directly reply to the replication study. While we cannot guarantee publications, we created 3ie’s Replication Paper Series (RPS) to partially address both of these concerns. The RPS provides an outlet for the replicating researchers to publish their work, and for original authors to respond to it. We view the RPS as a repository of replication research, including confirmatory studies that might struggle to find space in a journal.

If you’re interested in replication research, here are a few ways to get involved with 3ie’s replication programme:

– We’re planning another replication window. Send us the titles of recently published policy relevant development impact evaluations papers that you think should be considered for future replication to replication@3eimpact.org.

– Apply for a replication award when we open our next window.

– Volunteer to serve as an external reviewer of replication research.

– Submit your replication paper, even if it wasn’t funded by 3ie, to our RPS (here are the instructions).