In economic sciences, empirically-based studies have become increasingly important: According to Hamermesh (2012), the number of contributions to journals in which authors utilized self-collected or externally produced datasets for statistical analyses have massively increased in the course of the last decades.

With the growing relevance of publications based on empirical research, new questions and challenges emerge: Issues like integrating research data and scripts to run a data model in the broader context of a published article to foster replicable research and validation of scientific results are becoming increasingly important for both researchers and editors of scholarly journals.

Especially for a scientific discipline like economics, the effects of flawed research might have a huge impact on society, as the prominent debate of Reinhart’s and Rogoff’s “Growth in a time of debt” (2010) illustrated. Their paper attracted much attention and the results were cited by US vice presidential candidate Paul Ryan and EU monetary affairs commissioner Olli Rehn to justify austerity policy.

But when Rogoff and Reinhart provided the Excel-sheet of their calculations for teaching purposes to a student in 2013, this student, Thomas Herndon, was not able to replicate the results of the paper. Furthermore, he discovered that the Excel-sheet contained faulty calculations and selectively omitted data, which casted massive doubts on Reinhart’s and Rogoff’s findings. 

Despite the fact, that the paper of Rogoff and Reinhart has been published in the American Economic Review (AER), a journal having a strict data availability policy, the paper of the two American researchers has been exempted from this policy.

Therefore one could ask how journals in economics and business studies generally handle the challenges associated with empirically-based research. At least in theory, journals should serve as a quality assurance for economic research. On these grounds, peer-review was established to ensure a high quality of published research. But apparently, peer-review does not include the data appendices or other materials associated with empirically-based research: According to the US-economist B.D. McCullough, journals often fail to ensure the robustness of published results. After investigating the data archives of some economic journals, he concluded: „Despite claims that economics is a science, no applied economics journal can demonstrate that the results published in its pages are replicable, i.e., that there exist data and code that can reproduce the published results. No theory journal would dream of publishing a result (theorem) without a demonstration (proof) that the reader can trust the result.” (McCullough, 2009)   

To analyse how journals in economics and business studies deal with the challenge of reproducible research since the new decade, in 2010 we applied for funding from the German Research Foundation (DFG) for a project called “European Data Watch Extended – EDaWaX“. EDaWaX has several goals: The main objective of the project is to develop a software application for editors of social sciences’ journals. This software facilitates the management of publication-related research data. To gather some of the functional requirements for the development of the application, we analysed the number and specifications of existing data policies of economics journals for the first time in 2011. In 2014 we expanded our study. In our recent paper, we analysed the data policies of scholarly journals available in a sample of 346 journals. Many of them are among the top-journals of the profession. In contrast to our study in 2011, we also included a big share of journals in business studies to compare both branches of economic research.

Especially for economics journals we are able to state that things are changing slowly but steady: More than fourth of all economics journals in our sample are equipped with more or less functional data policies. While some journals pay lip-service to reproducible research, others effectively enforce their data policy.

In our paper we summarise the findings of this empirical study. We regard both the extent and the quality of journals’ data policies, which should facilitate replications of published empirical research. The paper presents some characteristics of journals equipped with data policies and gives some recommendations for suitable data policies in economics and business sciences journals. In addition, we also evaluate the journals’ data archives to roughly estimate whether these journals enforce data availability.

References:

Hamermesh, D.S. (2012), Six Decades of Top Economics Publishing: Who and How?, National Bureau of Economic Research, Working Paper 18635. Retrieved from http://www.peri.umass.edu/236/hash/31e2ff374b6377b2ddec04deaa6388b1/publication/566.

Herndon, T.; Ash, M. & Pollin R. (2013), Does High Public Debt Consistently Stifle Economic Growth? A Critique of Reinhart and Rogoff, Political Economy Research Institute. Retrieved from: http://www.peri.umass.edu/fileadmin/pdf/working_papers/working_papers_301-350/WP322.pdf.

Krugman, P. (2013), The Excel Depression, The New York Times, April 19 2013, p. A31. Retrieved from http://www.nytimes.com/2013/04/19/opinion/krugman-the-excel-depression.html?_r=4.

McCullough, B.D. (2009): Open Access Economics Journals and the Market for Reproducible Economic Research. Economic Analysis and Policy 39, 1, pp. 117-126.

Rehn, O. (2013), Letter to ECOFIN Ministers – ARES (2013)185796, February 13 2013, Retrieved from: http://ec.europa.eu/archives/commission_2010-2014/rehn/documents/cab20130213_en.pdf.

Reinhart, C.M. & Rogoff, K.S. (2010), Growth in a Time of Debt, American Economic Review 100, 2, 573–578. Retrieved from https://www.aeaweb.org/articles.php?doi=10.1257/aer.100.2.573.

Ryan, P. (2013), The Path to Prosperity: A Blueprint for American Renewal. Fiscal Year 2013 Budget Resolution, House Budget Committee. Retrieved from http://budget.house.gov/uploadedfiles/pathtoprosperity2013.pdf.

The American Economic Review (2005), Data Availability Policy. Retrieved from: https://www.aeaweb.org/aer/data.php.

Vlaeminck, S. / Herrmann L.K. (2015), Data policies and data archives: A new paradigm for academic publishing in economic sciences? In: Schmidt, B. & Dobreva, M. (Eds.), New Avenues for Electronic Publishing in the Age of Infinite Collections and Citizen Science: Scale, Openness and Trust. Proceedings of the 19th International Conference on Electronic Publishing, September 2015. Retrieved from doi:10.3233/978-1-61499-562-3-145.