“Next year, this topic should not be discussed in a pre-conference workshop but in the opening plenum of the conference!” This statement by a young researcher not only concluded the workshop but also gave bright prospects to replications in Economics.
On September 8, 2017 the ZBW Leibniz Information Center for Economics hosted the workshop “Replications in Empirical Economics – Ways out of the Crisis” at the Annual Conference of the Verein für Socialpolitik in Vienna, Austria. Thirty participants and four speakers engaged in lively and stimulating discussions about replications and the publication of replications in Economics. .
Hilmar Schneider, Director of the Institute of Labor Economics (IZA, Germany), made an enthusiastic plea for replications in economics and sharply criticized the credibility of much economic research. He argued that without replication research, knowledge can only grow horizontally rather than vertically. According to Schneider, the core problem lies in the pressure for novelty and original research that is exerted by the academic culture and the scientific publication system. As a consequence, pseudo-innovations are preferred to research that may actually have an impact on society. Arbitrary findings that lack verification are insufficient to properly inform and support policy decisions.
Christiane Joerk, Program Director from the German Research Foundation (DFG), summarized the DFG Statement on the Replicability of Research Results. The DFG already financially supports projects that strengthen the infrastructure for replications (e.g. IREE – see below) and will also fund replication studies in the future.
“Is it possible to publish replication studies in the International Journal for Re-Views in Empirical Economics using a pseudonym?” This question reflects the situation of young researchers, who are caught between good scientific practice and the current academic culture. Alarmingly, good scientific practice and academic culture apparently pose a conflict to researchers.
The wish for the anonymous publication of replication studies also reflects the bad reputation of replication studies in economics. On the one hand, most researchers state that replication studies are a critical part of scientific progress and indispensable for good scientific practice. On the other hand, as documented by Maren Duvendack from the University of East Anglia (UK), researchers who do replications are seen to be engaged in “bullying” and “persecution”, and referred to as “research parasites”. Together with her colleagues Richard Palmer Jones and Bob Reed she analyzed the publication market for replications in economics. The authors found that published replications in economics have been increasing since the mid-seventies. However the absolute number is still alarmingly low (with a peak of 22 published replications in economics in 2012). Compared to other disciplines such as psychology and political science, economics lags behind with respect to replication efforts. According to Duvendack, a key obstacle could be the lack of publication outlets for replication studies in economics.
As a partial solution to the replication crisis, Martina Grunow, Project Manager at the ZBW Leibniz Information Center for Economics (Germany), concluded the talks with her presentation of the International Journal for Re-Views in Empirical Economics (IREE). She summarized the incentive problems for authors to conduct replication studies and as a result a lack of publication possibilities for this kind of research. IREE is a peer-reviewed and open-access e-journal which is dedicated to the publication of replication studies in empirical economics. Replication studies are published without regard to their results. IREE-papers are made citable with DOIs and are internationally disseminated via EconStor and RePEc. As the project is funded by the German Research Foundation (DFG) and the ZBW Leibniz Information Center for Economics, IREE does not charge any publication fees. By providing a publication platform for replication studies, IREE aims to encourage the credibility of economics research based on robust and replicable findings.
With the prospect of publishing replication studies, doctoral students participating in the workshop expressed the view that replications should become a mandatory part of their dissertation research. They argued that by doing so, replication research would receive deserved credibility and not be buried under the pressure for pseudo-innovations. This is especially important at the beginning of their academic careers. The participants also agreed on the importance of teaching the value of replications to students.
Dr. Martina Grunow is Managing Editor of the International Journal for Re-Views in Empirical Economics (IREE) and is an associate researcher at the Canadian Centre for Health Economics (CCHE). She can be contacted by email at m.grunow@zbw.eu.
REFERENCES:
Duvendack, M., Palmer-Jones, R.W. & Reed, W.R., 2015. Replications in Economics: A Progress Report, Econ Journal Watch, 12(2): 164-191.
[From the article “Go Forth and Replicate: On Creating Incentives for Repeat Studies” by Michael Schulson at the web magazine Undark]
“Suggested reasons for the [replication] crisis are many. Some researchers have blamed the scientific publishing culture itself …. But some stakeholders also point to a more concrete and obvious root cause: a lack of direct incentives to replicate other researchers’ work, including very little funding to do replications. This raises some key questions, including whether the federal government — the largest single funder of basic science in the world — ought to be doing more to encourage and underwrite basic replication research. At the moment, only token reforms have been undertaken in the U.S., but there are signs that novel programs are gestating elsewhere, including a European pilot program that could serve as a model for replication funding in other countries.”
[From the blog by Justin Gallagher entitled “Public Data that Isn’t (or Wasn’t) Public” posted at BITSS]
“I recently completed a coauthored working paper, together with Paul J. Fisher, that examines whether electronic monitoring via red light traffic cameras leads to fewer vehicle accidents and injuries. As part of the project, we encountered the bizarre situation where researchers at The University of Texas at Austin, a flagship public state university, fought tooth and nail against releasing vehicle accident data originally collected by the Texas Department of Transportation (TxDOT).”
Replication can often be thought of as a useful tool to train graduate students or as a starting point for a new line of research, but sometimes replication is necessary as a means to check the robustness of results that can directly influence policy. Recently, Nunn and Qian (US Food Aid and Civil Conflict, American Economic Review 2014; 104: 1630–1666) found that United States (US) food aid increases the incidence and duration of civil conflict in recipient countries. This paper has received significant attention and has even been noticed by the United States Agency for International Development (USAID).
If the results of their study are robust, policymakers can attempt to minimize the predicted negative impacts. In our paper, we first were able to successfully replicate the results of Nunn and Qian (2014) using their data, but alternative software (R instead of Stata). We then attempted to further scrutinize one of their conclusions. Specifically, the authors claim that the adverse effect of US food aid on conflict does not vary across pre-determined characteristics of aid recipient countries (a seemingly strong assumption across sometimes vastly different nations). Nunn and Qian (2014) made attempts to allow for heterogeneity in their regression models by interacting US food aid with these pre-determined characteristics, but this simply amounted to group averages which may miss the underlying heterogeneity.
In order to check for more sophisticated forms of heterogeneity, we used a semiparametric estimation procedure. While the results visually suggested the presence of some amount of heterogeneity, this could not be determined statistically as we were unable to formally reject any of the parametric specifications in Nunn and Qian (2014). The conclusion of such a replication is that their models cannot be rejected using their data and we argue that the results of their paper are robust.
While we rightly criticize studies that cannot be replicated, we should also make note of those that can be replicated. It is typically a non-trivial task and while we were successful, we suggest that this study be further tested with samples from a different set of countries (both recipients and donors) and/or time periods.
Chi-Yang Chu is an assistant professor of economics at National Taipei University. Daniel J. Henderson is a professor of economics and the J. Weldon and Delores Cole Faculty Fellow at the University of Alabama. Le Wang is an associate professor of economics and the Chong K. Liew Chair in Economics at the University of Oklahoma. Correspondence about this blog should be directed to Daniel Henderson at djhender@culverhouse.ua.edu.
References
[1] N. Nunn and N. Qian, US Food Aid and Civil Conflict. American Economic Review. 104, 1630–1666 (2014)
[2] C.-Y. Chu, D. J. Henderson and L. Wang. The Robust Relationship between US Food Aid and Civil Conflict. Journal of Applied Econometrics. 32, 1027-1032 (2017)
[The following is taken from a blog by Hilda Bastian at the blogsite “Absolutely Maybe” at PLOS Blogs]
“As I’ve spent time with the badges “magic bullet” – simple! cheap! no side effects! dramatic benefits! – supported by a single uncontrolled study by an influential opinion leader, with a biased design in a narrow unrepresentative context, very small number of events, and short timeframe….I’ve come to think its biggest lesson may be that even many open science advocates have yet to fully absorb the implications of science’s reliability problems.”
In a recent post, TRN highlighted a recent working paper touting the benefits of something called an “R-Factor.” The R-Factor is a metric that would report — for each published empirical study — the reproducibility rate of that study in subsequent research. In a recent blog at Discover magazine’s website, Neuroskeptic blogs:
“A new tool called the R-factor could help ensure that science is reproducible and valid, according to a preprint posted on biorxiv: Science with no fiction. The authors, led by Peter Grabitz, are so confident in their idea that they’ve created a company called Verum Analytics to promote it. But how useful is this new metric going to be?”
Neuroskeptic’s answer: “Not very.” To read more, click here.
There is considerable concern among scholars that empirical papers face a drastically smaller chance of being published if the results looking to confirm an established theory turn out to be statistically insignificant. Such a publication bias can provide a wrong picture of economic magnitudes and mechanisms.
Against this background, the journal FinanzArchiv/Public Finance Analysis recently posted a call for papers for a special issue on “Insignificant Results in Public Finance”. The editors are inviting the submission of carefully executed empirical papers that – despite using state of the art empirical methods – fail to find significant estimates for important economic effects that have widespread acceptance.
It has been estimated that studies in economic behavioral research and psychology are ten times more likely to be published if they present statistically significant effects. Because a significant result may happen by chance, too much weight is attributed to them in the scientific literature. The associated publication bias can produce overestimates of the effectiveness of economic policy measures, psychological impacts or even medical medications.
While several ways to address this issue exist, a correction that is most directly related to the problem concerns the attitude of the scientific editors. Publication bias and the negative incentives for researchers are tackled at the root when studies are assessed on the basis of the methodology used and the quality of the data — and not on the results obtained. It requires a certain self-commitment of the journals to the increased publication of so-called “non-significant” results. Such a self-commitment was recently submitted by the editors of FinanzArchiv/Public Finance Analysis and is reflected in its call for papers.
The deadline for submissions to the special issue is 15 September 2017. Papers can be uploaded here: Submitting authors should indicate that their paper is being submitted to the special issue “Insignificant Results in Public Finance”. The editors would like to note that if any insignificant results transform into statistically significant results as an outcome of the refereeing process, this will not be held as an argument against publication. In this case, the paper may be shifted into a different issue of the Journal.
FinanzArchiv was first published in 1884, which makes it one of the world’s oldest professional journals in economics and the oldest journal of public finance. The current editors are Katherine Cuff, Ronnie Schöb and Alfons Weichenrieder. Within public economics, a strong focus is on topics as taxation, public debt, public goods, public choice, federalism, market failure, social policy and the welfare state.
Alfons Weichenrieder is Professor of Economics and Public Finance at Goethe-University Frankfurt and a guest research professor at the Institute of International Taxation of Vienna University of Economics and Business. He can be contacted via email at a.weichenrieder@em.uni-frankfurt.de.
[From the Center for Open Science webpage.]
“If you have a project that is entering the planning or data collection phase, we’d like you to try out a preregistration. Through our $1 Million Preregistration Challenge, we’re giving away $1,000 to 1,000 researchers who preregister their projects before they publish them. It’s straightforward to complete and will really enhance your research output.”
ScienceOpen has a collection entitled: “Remedies to the Reproducibility Crisis”. The collection is introduced thusly:
“Psychology, Medicine, Neuroscience and many other research fields, are facing a serious reproducibility crisis, that is, most of the findings published in peer-review journals, independently from their prestige, are not replicable. This collection aims at offering all remedies suggested to fix this problem.”
The collection currently consists of 24 articles on topics such as:
– “A manifesto for reproducibile science” (Munafo et al.)
– Scientific Standards: Promoting an open research culture (Nosek et al.)
– “The New Statistics: why and how” (Cumming)
– “Badges to acknowledge open practices: A simple, low-cost, effective method for increasing transparency” (Kidwell, et al.)
– “Calculating and reporting effect sizes to facilitate cumulative science” (Lakens)
– “The Peer Reviewers’ Openess Initiative: incentivizing open research practices through peer review” (Morey, et al.)
– “The influence of journal submission guidelines on authors’ reporting of statistics and use of open research practices” (Giofre et al.)
– “On the reproducibility of meta-analyses: six practical recommendations” (Lakens, Hilgard, and Staaks)
– “Equivalence Tests” (Lakens)
– “An agenda for purely confirmatory research” (Wagenmakers et al.)
– And more.
To see the collection and read more, click here.
QUICK: Does finding a significant effect when the sample size is small make it more likely that the effects are real and important? Or less?
James Heckman, Nobel Prize winning economist, says more:
“Also holding back progress are those who claim that Perry and ABC are experiments with samples too small to accurately predict widespread impact and return on investment. This is a nonsensical argument. Their relatively small sample sizes actually speak for — not against — the strength of their findings. Dramatic differences between treatment and control-group outcomes are usually not found in small sample experiments, yet the differences in Perry and ABC are big and consistent in rigorous analyses of these data.” (click here for source).
Andrew Gelman, Professor Statistics and Political Science at Columbia University, and blogger extraordinaire, says less:
“I agree with Stuart Buck that Heckman is wrong here. Actually, the smaller sample sizes (and also the high variation in these studies) speaks against—not for—the strength of the published claims.”
Who do YOU think is right?
To read more from Gelman, click here. His argument is elaborated in this working paper.
You must be logged in to post a comment.