Replication can often be thought of as a useful tool to train graduate students or as a starting point for a new line of research, but sometimes replication is necessary as a means to check the robustness of results that can directly influence policy. Recently, Nunn and Qian (US Food Aid and Civil Conflict, American Economic Review 2014; 104: 1630–1666) found that United States (US) food aid increases the incidence and duration of civil conflict in recipient countries. This paper has received significant attention and has even been noticed by the United States Agency for International Development (USAID).
If the results of their study are robust, policymakers can attempt to minimize the predicted negative impacts. In our paper, we first were able to successfully replicate the results of Nunn and Qian (2014) using their data, but alternative software (R instead of Stata). We then attempted to further scrutinize one of their conclusions. Specifically, the authors claim that the adverse effect of US food aid on conflict does not vary across pre-determined characteristics of aid recipient countries (a seemingly strong assumption across sometimes vastly different nations). Nunn and Qian (2014) made attempts to allow for heterogeneity in their regression models by interacting US food aid with these pre-determined characteristics, but this simply amounted to group averages which may miss the underlying heterogeneity.
In order to check for more sophisticated forms of heterogeneity, we used a semiparametric estimation procedure. While the results visually suggested the presence of some amount of heterogeneity, this could not be determined statistically as we were unable to formally reject any of the parametric specifications in Nunn and Qian (2014). The conclusion of such a replication is that their models cannot be rejected using their data and we argue that the results of their paper are robust.
While we rightly criticize studies that cannot be replicated, we should also make note of those that can be replicated. It is typically a non-trivial task and while we were successful, we suggest that this study be further tested with samples from a different set of countries (both recipients and donors) and/or time periods.
Chi-Yang Chu is an assistant professor of economics at National Taipei University. Daniel J. Henderson is a professor of economics and the J. Weldon and Delores Cole Faculty Fellow at the University of Alabama. Le Wang is an associate professor of economics and the Chong K. Liew Chair in Economics at the University of Oklahoma. Correspondence about this blog should be directed to Daniel Henderson at djhender@culverhouse.ua.edu.
References
[1] N. Nunn and N. Qian, US Food Aid and Civil Conflict. American Economic Review. 104, 1630–1666 (2014)
[2] C.-Y. Chu, D. J. Henderson and L. Wang. The Robust Relationship between US Food Aid and Civil Conflict. Journal of Applied Econometrics. 32, 1027-1032 (2017)
[The following is taken from a blog by Hilda Bastian at the blogsite “Absolutely Maybe” at PLOS Blogs]
“As I’ve spent time with the badges “magic bullet” – simple! cheap! no side effects! dramatic benefits! – supported by a single uncontrolled study by an influential opinion leader, with a biased design in a narrow unrepresentative context, very small number of events, and short timeframe….I’ve come to think its biggest lesson may be that even many open science advocates have yet to fully absorb the implications of science’s reliability problems.”
In a recent post, TRN highlighted a recent working paper touting the benefits of something called an “R-Factor.” The R-Factor is a metric that would report — for each published empirical study — the reproducibility rate of that study in subsequent research. In a recent blog at Discover magazine’s website, Neuroskeptic blogs:
“A new tool called the R-factor could help ensure that science is reproducible and valid, according to a preprint posted on biorxiv: Science with no fiction. The authors, led by Peter Grabitz, are so confident in their idea that they’ve created a company called Verum Analytics to promote it. But how useful is this new metric going to be?”
Neuroskeptic’s answer: “Not very.” To read more, click here.
There is considerable concern among scholars that empirical papers face a drastically smaller chance of being published if the results looking to confirm an established theory turn out to be statistically insignificant. Such a publication bias can provide a wrong picture of economic magnitudes and mechanisms.
Against this background, the journal FinanzArchiv/Public Finance Analysis recently posted a call for papers for a special issue on “Insignificant Results in Public Finance”. The editors are inviting the submission of carefully executed empirical papers that – despite using state of the art empirical methods – fail to find significant estimates for important economic effects that have widespread acceptance.
It has been estimated that studies in economic behavioral research and psychology are ten times more likely to be published if they present statistically significant effects. Because a significant result may happen by chance, too much weight is attributed to them in the scientific literature. The associated publication bias can produce overestimates of the effectiveness of economic policy measures, psychological impacts or even medical medications.
While several ways to address this issue exist, a correction that is most directly related to the problem concerns the attitude of the scientific editors. Publication bias and the negative incentives for researchers are tackled at the root when studies are assessed on the basis of the methodology used and the quality of the data — and not on the results obtained. It requires a certain self-commitment of the journals to the increased publication of so-called “non-significant” results. Such a self-commitment was recently submitted by the editors of FinanzArchiv/Public Finance Analysis and is reflected in its call for papers.
The deadline for submissions to the special issue is 15 September 2017. Papers can be uploaded here: Submitting authors should indicate that their paper is being submitted to the special issue “Insignificant Results in Public Finance”. The editors would like to note that if any insignificant results transform into statistically significant results as an outcome of the refereeing process, this will not be held as an argument against publication. In this case, the paper may be shifted into a different issue of the Journal.
FinanzArchiv was first published in 1884, which makes it one of the world’s oldest professional journals in economics and the oldest journal of public finance. The current editors are Katherine Cuff, Ronnie Schöb and Alfons Weichenrieder. Within public economics, a strong focus is on topics as taxation, public debt, public goods, public choice, federalism, market failure, social policy and the welfare state.
Alfons Weichenrieder is Professor of Economics and Public Finance at Goethe-University Frankfurt and a guest research professor at the Institute of International Taxation of Vienna University of Economics and Business. He can be contacted via email at a.weichenrieder@em.uni-frankfurt.de.
[From the Center for Open Science webpage.]
“If you have a project that is entering the planning or data collection phase, we’d like you to try out a preregistration. Through our $1 Million Preregistration Challenge, we’re giving away $1,000 to 1,000 researchers who preregister their projects before they publish them. It’s straightforward to complete and will really enhance your research output.”
ScienceOpen has a collection entitled: “Remedies to the Reproducibility Crisis”. The collection is introduced thusly:
“Psychology, Medicine, Neuroscience and many other research fields, are facing a serious reproducibility crisis, that is, most of the findings published in peer-review journals, independently from their prestige, are not replicable. This collection aims at offering all remedies suggested to fix this problem.”
The collection currently consists of 24 articles on topics such as:
– “A manifesto for reproducibile science” (Munafo et al.)
– Scientific Standards: Promoting an open research culture (Nosek et al.)
– “The New Statistics: why and how” (Cumming)
– “Badges to acknowledge open practices: A simple, low-cost, effective method for increasing transparency” (Kidwell, et al.)
– “Calculating and reporting effect sizes to facilitate cumulative science” (Lakens)
– “The Peer Reviewers’ Openess Initiative: incentivizing open research practices through peer review” (Morey, et al.)
– “The influence of journal submission guidelines on authors’ reporting of statistics and use of open research practices” (Giofre et al.)
– “On the reproducibility of meta-analyses: six practical recommendations” (Lakens, Hilgard, and Staaks)
– “Equivalence Tests” (Lakens)
– “An agenda for purely confirmatory research” (Wagenmakers et al.)
– And more.
To see the collection and read more, click here.
QUICK: Does finding a significant effect when the sample size is small make it more likely that the effects are real and important? Or less?
James Heckman, Nobel Prize winning economist, says more:
“Also holding back progress are those who claim that Perry and ABC are experiments with samples too small to accurately predict widespread impact and return on investment. This is a nonsensical argument. Their relatively small sample sizes actually speak for — not against — the strength of their findings. Dramatic differences between treatment and control-group outcomes are usually not found in small sample experiments, yet the differences in Perry and ABC are big and consistent in rigorous analyses of these data.” (click here for source).
Andrew Gelman, Professor Statistics and Political Science at Columbia University, and blogger extraordinaire, says less:
“I agree with Stuart Buck that Heckman is wrong here. Actually, the smaller sample sizes (and also the high variation in these studies) speaks against—not for—the strength of the published claims.”
Who do YOU think is right?
To read more from Gelman, click here. His argument is elaborated in this working paper.
In a recent working paper (“Science with no fiction: measuring the veracity of scientific reports by citation analysis”), Peter Grabitz, Yuri Lazebnik, Josh Nicholson, and Sean Rife suggest that one solution to the “crisis” in scientific credibility is publication of an article’s “R-Factor”. To calculate the R-Factor for a given study, one would comb through all the papers that cite a given study, then count up the number of attempts to confirm the findings from the original study. The R-Factor is simply the ratio of confirming studies over total attempts. R-Factors close to 1 indicate a study is likely to be true. R-Factors close to 0, not so much. The authors give an example from three studies in biomedical research. And how would this be done for thousands and thousands of studies, with results being continuously updated? The authors suggest this could be done through machine learning technology.
The main difference between “registered reports” and “results-free peer reviews” is timing of data analysis. With registered reports, plans are registered and reviewed before data are collected and analyzed. With results-free peer reviews, everything is completed, but the reviewers are blinded to the results. In this blog post, Saloni Krishnan discusses his experience with results-free reviewing and weighs the pros and cons versus registered reports:
“I just had my first pre-registered and results-free peer review paper published at BMC Psychology (Krishnan, Watkins, & Bishop, 2017, BMC Psychology). There’s more about why the journal is trying this format here. It’s a brave new world of open science out there, and I thought it’s worth trying to have some discussion about how this process differs from a standard submission, and whether it’s worth trying. I’ve consequently summarised my experience of this format, and a few takeaways for future pre-registered studies.”
SALONI KRISHNAN
[NOTE: This post refers to the article “An Economic Approach to Alleviate the Crises of Confidence in Science: With an Application to the Public Goods Game” by Luigi Butera and John List. The article is available as a working paper which can be downloaded here.]
In the process of generating scientific knowledge, scholars sometimes stumble upon new and surprising results. Novel studies typically face a binary fate: either their relevance and validity is dismissed, or findings are embraced as important and insightful. Such judgements however commonly rely on statistical significance as the main criterion for acceptance. This poses two problems, especially when a study is the first of its kind.
The first problem is that novel results may be false positives simply because of the mechanics of statistical inference. Similarly, new surprising results that suffer from low power, or marginal statistical significance, may sometimes be dismissed even though they point toward an economic association that is ultimately true.
The second problem has to do with how people should update their beliefs based on unanticipated new scientific evidence. Given the mechanics of inference, it is difficult to provide a definite answer when such evidence is based on one single exploration. To fix ideas, suppose that before running an experiment, a Bayesian scholar had a prior about the likelihood of a given result being true of only 1%. After running the experiment and observing the significant results (significant at, say, 5% level), the scholar should update his beliefs to 13.9%, a very large increase relative to the initial beliefs. Posterior beliefs can be easily computed, for any given prior, by dividing the probability that a true result is declared true by the probability that any result is declared true. Even more dramatically, a second scholar who for instance had priors of 10%, instead of 1%, would update his posterior beliefs to 64%. The problem is clear: posterior beliefs generated from low priors are extremely volatile when they only depend on evidence provided by a single study. Finding a referee with priors of 10% or 1% can make or break a paper!
The simple solution to this problem is of course to replicate the study: as evidence accumulates, posterior beliefs converge. Unfortunately, the incentives to replicate existing studies are rarely in place in the social sciences: once a paper is published, the original authors have little incentive to replicate their own work. Similarly, the incentives for other scholars to closely replicate existing work are typically very low.
To address this issue, we proposed in our paper a simple incentive-compatible mechanism to promote replications, and generate mutually beneficial gains from trade between scholars. Our idea is simple: upon completion of a study that reports novel results, the authors make it available online as a working paper, but commit never to submit it to a peer-reviewed journal for publication. They instead calculate how many replications they need for beliefs to converge to a desired level, and then offer co-authorship for a second, yet to be written, paper to other scholars willing to independently replicate their study. Once the team of coauthors is established, but before replications begin, the first working paper is updated to include the list of coauthors and the experimental protocol is registered at the AEA RCT registry. This guarantees that all replications, both failed and successful, are accounted for in the second paper. The second paper will then reference the first working paper, include all replications, and will be submitted to a peer-reviewed journal for publication.
We put our mechanism to work on our own experiment where we asked: can cooperation be sustained over time when the quality of a given public good cannot be precisely estimated? From charitable investments to social programs, uncertainty about the exact social returns from these investments is a pervasive characteristic. Yet we know very little about how people coordinate over ambiguous and uncertain social decisions. Surprisingly, we find that the presence of (Knightian) uncertainty about the quality of a public good does not harm, but rather increases cooperation. We interpret our finding through the lenses of conditional cooperation: when the value of a public good is observed with noise, conditional cooperators may be more tolerant to observed reductions in their payoffs, for instance because such reductions may be due, in part, to a lower-than-expected quality of the public good itself rather than solely to the presence of free-riders. However, we will wait until all replications are completed to draw more informed inference about the effect of ambiguity on social decisions.
One final note: while we believe that replications are always desirable, we do not by any means suggest that all experiments, lab or field, necessarily need to follow our methodology. We believe that our approach is best suited for studies that find results that are unanticipated, and in some cases at odds with the current state of knowledge on a topic. This is because in these cases, priors are more likely to be low, and perhaps more sensitive to other factors such as the experience or rank of the investigator. As such, we believe that our approach would be particularly beneficial for scholars at the early stages of their careers, and we hope many will consider joining forces together.
Luigi Butera is a Post-Doctoral scholar in the Department of Economics at the University of Chicago. He can be contacted via email at lbutera@uchicago.edu.
The Replication Network 
You must be logged in to post a comment.