In support of recent efforts by social scientists to address the ‘reproducibility crisis^’, the Journal of Development Effectiveness (JDEff) recently devoted a special issue on replication research studies in its last issue of 2019.  Most journals continue to favor new research rather than replication work and to publish research whose data and codes have not been tested. As editors of the journal, and as the past and current executive director of the International Initiative for Impact Evaluation (3ie) which hosts it, we felt that such an issue could increase awareness among funders and researchers of how replication strengthens the reliability, rigor and relevance of their investment.  It would also ensure that the replication research will be acknowledged and appreciated by the larger development community.

The special issue was devoted to the topic of enhancing financial inclusion in developing countries.  3ie, which has championed replication research for many years, had worked closely with the Bill and Melinda Gates Foundation’s Financial Services for the Poor (FSP) program to identify the studies, screen the applicants, and quality assure the replication research in this important area.  FSP invests millions of dollars to broaden the reach of low-cost digital financial services for the poor by supporting the most catalytic approaches to financial inclusion, such as the development of digital payment systems, advancing gender equality, and supporting national and regional strategies.  In doing so, it relies heavily on research and evidence studies, many of which, although cited and referenced heavily, have not been replicated.

We hope that these replications can be used appropriately by FSP and other stakeholders to inform future investments in an important part of the development toolkit – expanding financial services to the poor.  About 1.7 billion people worldwide are excluded from formal financial services, such as savings, payments, insurance, and credit. In developing economies, nearly one billion are still left out of the formal financial system, and there is a 9 percent persistent gender gap in financial inclusion in developing economies. 

Most poor households instead, operate almost entirely through a cash economy. They are cut off from potentially stabilizing and uplifting opportunities like building credit or getting a loan to start a business. And it’s harder to weather common financial setbacks, such as serious illness, a poor harvest, or an economic downturn, such as the one the world experiencing now due to the coronavirus epidemic.  In fact, just last week, one of us, who had just moved from Delhi to Manila, was able to help his former Indian housekeeper with a cash transfer, only because she had a bank account.  It took all of 15 minutes to send her much needed financial support, which would have been much more difficult otherwise.  Millions of others have no access to such mechanisms.

The issue in JDEff replicates 6 important financial inclusion studies.  These include a study on providing banking access to farmers; three studies that evaluated interventions to introduce innovative alternatives to traditional banking, such as using mobile phones or biometric smartcards as payment mechanisms for transfers; and two studies that studied the effects of different kinds of transfers (cash versus kind; conditional versus unconditional) that are distributed through financial institutions.  Importantly, the replications were able to reproduce the principal results of all of the studies.  It is as important to highlight this finding, as much as it is to get notoriety through “gotcha” replications that appear to overturn results, such as in the “worm wars” of a few years ago.

The replications also make several useful findings about nuances that the original research may have missed, such as about heterogeneous effects.

Research must meet the higher bar of being good enough for decision-making that affects human lives (not merely good enough for publication. Organizations like 3ie which consider replication as an important tool in making research more rigorous takes the following lessons from this JDEff issue on doing replications in the future.  One is to ensure that, beyond taking the original research at face value, enough attention is dedicated to results not reported (to avoid reporting bias), to the policy significance of the reported results, to reflections about possible rival explanations for the results, and to how the main variables were constructed. Another replication deficiency in current research relates to the replication of qualitative research. While there is increasing acceptance that replication of quantitative research is part of best practice by funders and journals, replication in the qualitative research field is nascent. In a new initiative, 3ie is partnering with the Qualitative Data Repository (Syracuse University) to get their assistance in archiving and sharing select qualitative data, learn from the experience and thereby contribute to lessons and guidance on how to do this in the future.  Finally, within the evidence architecture, it is worthwhile promoting systematic reviews as a set of replications of studies in differentiated real life settings.

Marie Gaarder is the current Executive Director of the International Initiative for Impact Evaluation (3ie).  Emmanuel Jimenez is a Senior Fellow at and former Executive Director of 3ie, and Editor-in-Chief of the Journal of Development Effectiveness.  They can be contacted, respectively, at mgaarder@3ieimpact.org and ejimenez@3ieimpact.org.