This post is based on a keynote presentation I gave at the Editor’s Meeting of the International Journal for Re-Views of Empirical Economics in June 2020. It loosely follows up two previous attempts to summarize the state of replications in economics: (i) An initial paper by Maren Duvendack, Richard Palmer-Jones, and myself entitled “Replications in Economics: A Progress Report”, published in Econ Journal Watch in 2015; and (ii) a blog I wrote for The Replication Network (TRN) entitled “An Update on the Progress of Replications in Economics”, posted in October 2018.

In this instalment, I address two issues:

– Are there more replications in economics than there used to be?

– Which journals publish replications?

Are there more replications in economics than there used to be?

Before we count replications, we need to know what we are counting. Researchers use different definitions of replications, which produce different numbers. For example, at the time of this writing, Replication Wiki reports 670 replications at their website. In contrast, TRN, which relies heavily on Replication Wiki, lists 491 replications.

Why the difference? TRN employs a narrower definition of a replication. Specifically, it defines a replication as “any study published in a peer-reviewed journal whose main purpose is to determine the validity of one or more empirical results from a previously published study.”

Replications come in many sizes and shapes. For example, sometimes a researcher will develop a new estimator and want to see how it compares with another estimator. Accordingly, they replicate a previous study using the new estimator. An example is De Chaisemartin & d’Haultfoeuille’s “Fuzzy differences-in-differences” (Review of Economic Studies, 2018). D&H develop a DID estimator that accounts for heterogeneous treatment effects when the rate of treatment changes over time. To see the difference it makes, they replicate Duflo (2001) which uses a standard DID estimator.

Replication Wiki counts D&H as a replication. TRN does not. The reason TRN does not count D&H as a replication is because the main purpose of D&H is not to determine whether Duflo (2001) is correct. The main purpose of D&H is to illustrate the difference their estimator makes. This highlights the grey area that separates replications from other studies.

Reasonable people can disagree about the “best” definition of replication. I like TRN’s definition because it restricts attention to studies whose main goal is to determine “the truth” of a claim by a previous study. Studies that meet this criterion tend to be more intensive in their analysis of the original study and give it a more thorough empirical treatment. A further benefit is that TRN has consistently applied the same definition of replication over time, facilitating time series comparisons.

FIGURE 1 shows the growth in replications in economics over time. The graph is somewhat misleading because 2019 was an exceptional year, driven by special replication issues at the Journal of Development Studies, the Journal of Development Effectiveness, and, especially, Energy Economics. In contrast, 2020 will likely end up having closer to 20 replications. Even ignoring the big blip in 2019, it is clear that there has been a general upwards creep in the number of replications published in economics over time. It is, however, a creep, and not a leap. Given that there are approximately 40,000 articles published annually in Web of Science economics journals, the increase over time does not indicate a major shift in how the economics discipline values replications.

Which journals publish replications?

TABLE 1 reports the top 10 economics journals in terms of total number of replications published in their journal lifetimes. Over the years, a consistent leader in the publishing of replications has been the Journal of Applied Econometrics. In second place is the American Economic Review. However, an important distinction between these two journals is that JAE publishes both positive and negative replications; that is, replications that both confirm and refute the original studies. In contrast, the AER only very rarely publishes a positive replication.

There have been several new initiatives by journals to publish replications. Notably, the International Journal for Re-Views of Empirical Economics (IREE) was started in 2017 and is solely dedicated to the publishing of replications. It is an open access journal with no author processing charges (APCs), supported by a consortium of private and public funders. As of January 2021, it had published 10 replication studies.

To place the numbers in TABLE 1 in context, there are approximately 400 mainline economics journals. About one fourth (96) have ever published a replication. 2 journals account for approximately 25% of all replications that have ever been published. 9 journals account for over half of all replication studies. Only 25 journals (about 6% of all journals) have ever published more than 5 replications in their lifetimes.

Conclusion

While a little late to the party, economists have recently made noises about the importance of replication in their discipline. Notably, the 2017 Papers and Proceedings issue of the American Economic Review prominently featured 8 articles addressing various aspects of replications in economics. And indeed, there has been an increase in the number of replications over time. However, the growth in replications is best described as an upwards creep rather than a bold leap.

Perhaps the reason replications have not really caught on is because fundamental questions about replications have not been addressed. Is there a replication crisis in economics? How should “replication success” be measured? What is the “success rate” of replications in economics? How should the results of replications be interpreted? Do replications have a unique role to play in contributing to our understanding of economic phenomena? I take these up in subsequent instalments of this blog (to read the next instalment, click here).

Bob Reed is a professor of economics at the University of Canterbury in New Zealand. He is also co-organizer of the blogsite The Replication Network and Principal Investigator at UCMeta. He can be contacted at bob.reed@canterbury.ac.nz.