The Replication Network

When preparing to give a talk at a conference recently, I decided to update some information I had published a few years ago.  In McCullough (2009), I estimated that 16 economics journals had a mandatory data/code archive (archives that require only data do not support replication — see McCullough, McGeary and Harrison (2008)).  Vlaeminck (2013) counted 26 journals with a mandatory data/code archive. This is a non-trivial increase, since in 2004 only four economics journals had such a policy.  One might think that this increase bodes well for replicability in the economic science, but such is not the case.  It is all well and good to make data and code available for replications, but if there is no place for researchers to publish these replications, then all the mandatory data/code archives in the world will amount to only so much window dressing.

The problem is that editors do not want to admit that they publish unreplicable research, nor do they want to be bothered ensuring that the research they publish is replicable.  The fact is that very few journals will publish replications and the top-ranked journals only publish an infinitesimal number of replications.  Consequently, any editor is largely immune to the embarrassment that would arise if several of the articles he published were found to be not replicable.  Hence, editors have no incentive either to ensure the replicability of the articles they publish or to publish replications of the articles they do publish.  If researchers can’t get their replication articles published in decent journals, they won’t write the articles in the first place.  And this seems to be the present state of equilibrium, sub-optimal though it may be.  Worse, there seems to be a tacit collusion between the editors, in that one editor will not publish an article that exposes another editor as publishing unreplicable research.

Prima facie evidence of this sad state of affairs is the fact that Liebowitz’s failed replication of the JPE paper by Oberholzer-Gee and Strumpf still hasn’t been published, not by the the JPE and not be any other journal.  Anyone interested in replication should go to SRRN and read the papers by Liebowitz on this topic.  In “How Reliable is the Oberholzer-Gee and Strumpf Paper on File Sharing”, Liebowitz capably demonstrates fatal flaws in the data handling and analysis of the Oberholzer-Gee and Strumpf paper.  Actually, time is precious; just take my word for it so that you don’t have to read it: Liebowitz demolishes the Oberholzer-Gee/Strumpf paper.  In “Sequel to Liebowitz’s Comment on the Oberholzer-Gee and Strumpf Paper on File Sharing”, Liebowitz describes his efforts to get his paper published in the JPE.  This is the paper to be read. So Kafkaesque was Liebowitz’s ordeal that journalist Norbert Haring, writing in the German financial newspaper Handelsblatt (the German equivalent of the Wall Street Journal), said, “Steven Levitt, Editor of the Journal of Political Economy, uses a questionable tactic to block an undesired comment.   The subject of the criticised article was a hot topic.  On closer look, everything about the case was unusual.”  One might think that another journal with an interest in file sharing would publish Liebowitz’s paper….

No one can read these papers by Liebowitz and think that “truth will out” in the economics journals.  Yet there is cause for hope.

Third party organizations dedicated to replication have emerged in the past few years, such as 3ie (International Initiative for Impact Evaluation) and BITSS (Berkeley Initiative for Transparency in the Social Sciences) and EDAWAX (European Data Watch).  These organizations support replication without a necessary prospect of publication.  If these organizations can demonstrate that top journals are publishing non-replicable research, then the top journals might be embarrassed into admitting that their efforts to ensure replicability are insufficient.  And then Liebowitz’s article might finally get published.

References
===========
N. Haring, Handelsblatt, 23.06.2008

B. D. McCullough
“Open Access Economics Journals and the Market for Reproducible Economic Research”
Economic Analysis and Policy 39(1), 117-126, 2009

B. D. McCullough, Kerry Anne McGeary and Teresa D. Harrison
“Do Economics Journal Archives Promote Replicable Research?”
Canadian Journal of Economics 41(4), 1406-1420, 2008

Vlaeminck, Sven, 2013. “Data Management in Scholarly Journals and Possible Roles for Libraries – Some Insights from EDaWaX,” EconStor Open Access Articles, ZBW – German National Library of Economics, pages 49-79.