[This post is based on a presentation by Bob Reed at the Workshop on Reproducibility and Integrity in Scientific Research, held at the University of Canterbury, New Zealand, on October 26, 2018]

In 2015, Duvendack, Palmer-Jones, and Reed (DPJ&R) published a paper entitled “Replications in Economics: A Progress Report”. In that paper, the authors gave a snapshot of the use of replications in economics.

A little over three and a half years have passed since the research for that paper was completed. During that time, there has been much talk about the so-called “replication crisis”, including featured articles in the 2017 Papers and Proceedings issue of the American Economic Review. That issue spotlighted 8 articles addressing various aspects of replications in economics. Which raises the question, has anything changed since DPJ&R published their article?

In this blog, I update DPJ&R’s research. I focus on four measures of the use of replications in economics:

– Total number of replications published in economics journals

– Which journals say they will publish replications

– Which journals actually publish replications

– Which journals require authors of empirical papers to supply their data and code. And, of those, which journals actually do it.

Total Number of Replications Published in Economics Journals

DPJ&R defined a replication as any study published in a peer-reviewed journal whose main purpose was to verify a previously published study. Based upon that definition, the figure below reports the total number of replications published in economics journals over time. The solid, vertical, black line delineates the time period included in DPJ&R’s study.

At the time DPJ&R wrote their article, it looked like replications were “taking off”, with the publication of replications increasing exponentially. In the three and half years since, that impression needs to be moderated. While the publication of replications have definitely increased since the early 2000s, it would appear that the rate of publication has leveled off. Increasing talk about replications in economics has not been matched by a corresponding increase in the number of published replications.

Journals That Say They Publish Replications

In order to gauge the receptivity of journals to publish replications, DPF&R went to the websites of all the journals listed by Web of Science as “Economics” journals. At the time of their study, there were 333 such journals. The websites of 10 of these explicitly mentioned that they published replications. These are listed on the left hand side of the table below.

In August of this year, I and a team of students rechecked the websites of “Economics” journals listed by Web of Science – now totalling 360 journals. A total of 14 journals now explicitly state on their websites that they publish replications. Interestingly, the net gain of 4 journals is the result of the addition of 6 journals that have newly stated they will publish replications, minus two journals that have removed mention of publishing replications: Economic Development and Cultural Change and the International Journal of Forecasting no longer explicitly state a policy of publishing replications. 

Journals That Publish Replications

There are many journals that do not explicitly state they publish replications, but for which revealed preference shows they do. The left hand side of the table below reports published replications by journal for the time period covered by DPF&R. Note that the total of 206 published replications exceeds the number reported by DPJ&R. This is because I updated their list of replication studies and found additional studies that they missed. Through 2014, I identify a total of 21 journals that published more than 1 replication over their history. 5 journals were responsible for publishing approximately half of all replication studies, with the Journal of Applied Econometrics and the American Economic Review leading the pack.

Approximately three and a half years later, 26 journals have published more than 1 replication study. 5 journals still account for half of all published replication studies, with the same two journals leading the list. Notably, over half of the 71 replication studies published since 2014 have appeared in three journals: the Journal of Applied Econometrics, Econ Journal Watch, and Public Finance Review. The increase in Public Finance Review’s replications can be attributed to the fact that they introduced a dedicated replication section.

Journals That Require Authors to Provide Data and Code

In their article, DPJ&R surveyed the list of Web of Science “Economics” journals that “regularly” provided data and code for their empirical articles. “Regularly” was defined as providing data and code for at least half of the journal’s empirical articles in recent issues. They reported 27 articles met this criterion.

We updated the list of 360 Web of Science “Economics” journals and repeated the analysis. Only this time, we also kept track of which journals required authors to supply their data and code. A total of 41 journals explicitly stated that authors of empirical papers were required to provide data and code sufficient to replicate the results in the paper. Journals that said they “encouraged” authors to provide data and code, or that required authors to make their data and code available “upon request”, were not included in this list.

We then went through 15 of each journal’s most recently published empirical articles. If at least half (8) had both data and code, we classified it as satisfying the journal’s requirement. 23 of the 41 (56%) met this criterion. These are listed in the left hand side of the table below.

If less than half of the articles were accompanied by data and code, or if only data and not code were provided, or code and not data, we judged it to have not satisfied the criteria. In this manner, 18 of the 41 journals (44%) were determined to not be in compliance with their stated data and code policy. These are listed on the right hand side of the table.

Please note, however, an important caveat: most journals make an exception for confidential data. These could be data that were provided with strict confidentiality requirements, or subscription data where the vendor does not want to make the data public. We had no way of knowing if this was the reason an article did not provide data and code. Thus, some of the journals on the right hand side of the table may be in compliance with their data and code policy once one accounts for confidentiality restrictions.

Conclusion

Putting the above together, what can one make of the current status of replications in economics? Based upon what one sees in the journals, this is a case of water in the glass. For those who are optimists, the glass may be seen as half full. More journals are publishing more replication studies now than they were a decade ago. More journals are announcing that they publish replications. And more journals (one more!) are posting data and code along with their empirical articles. Replications and reproducibility are inching their way forward in the economics discipline.

However, for those who think there is a replication crisis in economics, the glass is half empty, and arguably not even that. The situation is a far cry from what is happening in some other social sciences, particularly psychology. There, articles like “Making Replication Mainstream” speak to a major culture change that seems to have been embraced by many, if not most, editors of leading journals in that discipline.

Some would argue that the reason psychology has been so willing to embrace replication is because that discipline has been more prone to questionable research practices. While that may be the case, the fact is, nobody really knows. There is only way to find out just how good, or bad, things are in economics. And that is to do more replications. Based on the above, it will be sometime yet before enough replications are done so that we can have a better idea of the status of reproducibility in the economics discipline.

Bob Reed is a professor of economics at the University of Canterbury in New Zealand. He is also co-organizer of the blogsite The Replication Network. He can be contacted at bob.reed@canterbury.ac.nz.