[From the abstract of the article “What Meta-Analyses Reveal About the Replicability of Psychological Research” by T.D. Stanley, Evan Carter, and Hristos Doucouliagos, published in Psychological Bulletin]
“Can recent failures to replicate psychological research be explained by typical magnitudes of statistical power,bias or heterogeneity? A large survey of 12,065 estimated effect sizes from 200 meta-analyses and nearly 8,000 papers is used to assess these key dimensions of replicability.”
“First, our survey finds that psychological research is, on average, afflicted with low statistical power. The median of median power across these 200 areas of research is about 36%, and only about 8% of studies have adequate power (using Cohen’s 80% convention). Second, the median proportion of the observed variation among reported effect sizes attributed to heterogeneity is 74% (I2). Heterogeneity of this magnitude makes it unlikely that the typical psychological study can be closely replicated when replication is defined as study-level null hypothesis significance testing.”
“… the low power and high heterogeneity that our survey finds fully explain recent difficulties to replicate highly regarded psychological studies and reveal challenges for scientific progress in psychology.”
[From the post “A study fails to replicate, but it continues to get referenced as if it had no problems. Communication channels are blocked.” by Andrew Gelman at Statistical Modeling, Causal Inference, and Social Science]
“In 2005, Michael Kosfeld, Markus Heinrichs, Paul Zak, Urs Fischbacher, and Ernst Fehr published a paper, ‘Oxytocin increases trust in humans.’ According to Google, that paper has been cited 3389 times.”
“In 2015, Gideon Nave, Colin Camerer, and Michael McCullough published a paper, ‘Does Oxytocin Increase Trust in Humans? A Critical Review of Research,’ where they reported:”
“Behavioral neuroscientists have shown that the neuropeptide oxytocin (OT) plays a key role in social attachment and affiliation in nonhuman mammals. Inspired by this initial research, many social scientists proceeded to examine the associations of OT with trust in humans over the past decade. . . . Unfortunately, the simplest promising finding associating intranasal OT with higher trust [that 2005 paper] has not replicated well.”
“…The paper’s only been out 3 years. Let’s look at recent citations, since 2017:”
– “Oxytocin increases trust in humans: 377 citations”
– “Does Oxytocin Increase Trust in Humans? A Critical Review of Research: 49 citations”
“…Just to be clear, I’m not saying the old paper should be getting zero citations. It may well have made an important contribution for its time, and even if its results don’t stand up to replication, it could be worth citing for historical reasons. But, in that case, you’d want to also cite the 2015 article pointing out that the result did not replicate.”
“The pattern of citations suggests that, instead, the original finding is still hanging on, with lots of people not realizing the problem.”
Last week I was treated to a great workshop titled “Reproducibility and Integrity in Scientific Research” at the University of Canterbury where I presented my article (joint with Benjamin D.K. Wood), “Which tests not witch hunts: A diagnostic approach for conducting replication research.” The article provides tips and resources for researchers seeking a neutral approach to replication research. In honor of the workshop and Halloween, I thought I’d scare up a blog post summarizing the article.
Why conduct replication research?
Suppose you’ve read a study that you consider to be innovative or influential. Why might you want to conduct a replication study of it? Here when I say ‘replication study’, I mean internal replication (or desk replication), for which the researcher uses the study’s original data to reassess the study’s findings. There are three reasons you might want to conduct such a study: to prove it right, to learn from it, or to prove it wrong. We rarely see the first reason stated, making it a bit of phantom. However, I am a big fan of conducting replication research to validate a study’s findings for the purpose of policy making or program design. We see the second reason – to learn from it – more often, although often in the context of graduate school courses on quantitative methods.
Instead, many fear that most replication studies are conducted with the desire to prove a study wrong. Zimmerman (2015) considers “turning replication exercises into witch hunts” to be an easy pitfall of replication research. Gertler, Galiani, and Romero (2018) report that unnamed third parties “speculated” that researchers for a well-known replication study sought to overturn results. The specter of speculation aside, why might replication researchers look for faults in a study?
One reason is publication bias. Experience shows that replication studies that question the results of original studies are more likely to be published, and Gertler, Galiani, and Romero (2018) provide evidence from a survey of editors of economics journals showing that editors are much more likely to publish a replication study that overturns results than one that confirms results. Regardless of publication bias, however, my experience funding replication studies while working at the International Initiative for Impact Evaluation (3ie) is that not all replication researchers carry torches and pitchforks. Many just don’t know where to start when conducting replication research. Without some kind of template or checklist to work from, these researchers are often haunted by the academic norm of critical review and approach their replication work from that standpoint.
To address this challenge, Ben Wood and I set out to develop a neutral approach to replication research based on elements of quantitative analysis and using examples from 3ie-funded replication studies. This approach is intended for researchers who want to dissect a study beyond just a pure replication (which is using the study’s methods and original data to simply reproduce the results in the published article). The diagnostic approach includes four categories: assumptions, data transformations, estimation, and heterogeneous outcomes.
Assumptions
The application of methods and models in conducting empirical research always involves making assumptions. Often these assumptions can be tested using the study data or using other data. Since my focus is often development impact evaluation, the assumptions I see most often are those supporting the identification strategy of a study. Examples include assuming no randomization failure in the case of random-assignment designs or assuming unobservables are time invariant in the case of difference-in-difference designs. Many other assumptions are also often necessary depending on the context of the research. For example, when looking at market interventions, researchers often assume that agents are small relative to the market (i.e., price takers). Even if the study data cannot be used to shed light on these assumptions, there may be other data that can.
In the Whitney, Cameron, and Winters (2018) replication study of the Galiani and Schargrodsky (2010) impact evaluation of a property rights policy change in Buenos Aires, the replication researchers note that the original authors provide balance tables for the full sample of 1,082 parcels but only conduct their analysis on a subset of 300 parcels. Whitney, et al. test the pre-program balance between program and comparison parcels on four key characteristics for the households in the analysis subset and find statistically significant differences for three of the four. Their further tests reveal that these imbalances do not change the ultimate findings of the study, however.
Data transformations
There is a lot of hocus pocus that goes into getting data ready for analysis. These spells determine what data are used, including decisions about whether to kill outliers, how to bring missing values back from the dead, and how to weight observations. We also often engage in potion making when we use data to construct new variables, including variables like aggregates (e.g., income and consumption) and indexes (e.g., empowerment and participation). Replication researchers can use the study data and sometimes outside data in order to answer questions about whether these choices are well supported and whether they make a difference to the analysis.
Kuecken and Valfort (2018) question the decision by Reinikka and Svensson (2005) to exclude certain schools from the analysis dataset used for their study of how an anti-corruption newspaper campaign affects enrollment and learning. The original study includes a footnote that the excluded schools experienced reductions in enrollment due to “idiosyncratic shocks”, which the original authors argue should not be systematically correlated with the explanatory variable. Kuecken and Valfort resurrect the excluded schools and find that the published statistical significance of the findings is sensitive to the exclusion.
Estimation methods
There are two sets of replication questions around estimation methods. One is whether different methods developed for similar statistical tasks produce the same results. A well-known example is the replication study conducted by epidemiologists Aiken, Davey, Hargreaves, and Hayes (2015) (published as two articles) of an impact evaluation of a health intervention conducted by economists Miguel and Kremer (2004). This replication study combined with systematic review evidence resulted in the worm wars, which were indeed spine-chilling. The second set of questions is how sensitive (or robust) the results are to parameters or other choices made when applying estimation methods. Many published studies include some sensitivity tests, but there are sometimes additional sensitivity tests that can be conducted.
Korte, Djimeu, and Calvo (2018) do the converse of worm wars – they apply econometric methods to data from an epidemiology trial by Bailey, et al. (2007) testing whether male circumcision reduces incidence of HIV infection. For example, Korte, et al. exploit the panel nature of the data, that is, repeated observations of the same individuals over time, by running a fixed effects model, which controls for unobserved individual differences that don’t change over time. They find that the econometric methods produce very similar results as the biostatistical methods for the HIV infection outcome, but produce some different results for the tests of whether male circumcision increases risky sexual behavior.
Heterogeneous outcomes
Understanding whether the data from a published study point to heterogeneous outcomes can be important for using the study’s findings for program design or policy targeting. These further tests on a study’s data are likely to be exploratory rather than confirmatory. For example, one might separate a random-assignment sample into men and women for heterogeneous outcomes analysis even if the randomization did not occur for these two groups separately. Exploration of heterogeneous outcomes in a replication study should be motivated by theoretical or clinical considerations.
Wood and Dong (2018) re-examine an agricultural commercialization impact evaluation conducted by Ashraf, Giné, and Karlan (2009). The commercialization program included promoting certain export crops and making it easier to sell all crops. The original study explores heterogeneous outcomes by whether the sample farmers grew the export crops before the intervention or not and find that those who did not grow these crops are more likely to benefit. Wood and Dong use value chain theory to hypothesize that the benefits of the program come from bringing farmers to the market, that is getting them to sell any crops (domestic or export). They look at heterogeneous outcomes by whether farmers grew any cash crops before the program and find that only those who did not grow cash crops benefit from the program.
Internal replication research provides validation of published results, which is especially important when those results are used for policy making and program design (Brown, Cameron, and Wood, 2014). It doesn’t need to be scary, and original authors don’t need to be spooked. The “which tests not witch hunts” paper provide tips and resources for each of the topics described above. The paper also provides a list of “don’ts” for replication research, which I’ll summarize in a separate post. Happy Halloween!
Annette N. Brown, PhD is Principal Economist at FHI 360, where she leads efforts to increase and enhance evidence production and use across all sectors and regions. She previously worked at 3ie, where she directed the research transparency programs, including the replication program.
[This post is based on a presentation by Bob Reed at the Workshop on Reproducibility and Integrity in Scientific Research, held at the University of Canterbury, New Zealand, on October 26, 2018]
In 2015, Duvendack, Palmer-Jones, and Reed (DPJ&R) published a paper entitled “Replications in Economics: A Progress Report”. In that paper, the authors gave a snapshot of the use of replications in economics.
A little over three and a half years have passed since the research for that paper was completed. During that time, there has been much talk about the so-called “replication crisis”, including featured articles in the 2017 Papers and Proceedings issue of the American Economic Review. That issue spotlighted 8 articles addressing various aspects of replications in economics. Which raises the question, has anything changed since DPJ&R published their article?
In this blog, I update DPJ&R’s research. I focus on four measures of the use of replications in economics:
– Total number of replications published in economics journals
– Which journals say they will publish replications
– Which journals actually publish replications
– Which journals require authors of empirical papers to supply their data and code. And, of those, which journals actually do it.
Total Number of Replications Published in Economics Journals
DPJ&R defined a replication as any study published in a peer-reviewed journal whose main purpose was to verify a previously published study. Based upon that definition, the figure below reports the total number of replications published in economics journals over time. The solid, vertical, black line delineates the time period included in DPJ&R’s study.
At the time DPJ&R wrote their article, it looked like replications were “taking off”, with the publication of replications increasing exponentially. In the three and half years since, that impression needs to be moderated. While the publication of replications have definitely increased since the early 2000s, it would appear that the rate of publication has leveled off. Increasing talk about replications in economics has not been matched by a corresponding increase in the number of published replications.
Journals That Say They Publish Replications
In order to gauge the receptivity of journals to publish replications, DPF&R went to the websites of all the journals listed by Web of Science as “Economics” journals. At the time of their study, there were 333 such journals. The websites of 10 of these explicitly mentioned that they published replications. These are listed on the left hand side of the table below.
In August of this year, I and a team of students rechecked the websites of “Economics” journals listed by Web of Science – now totalling 360 journals. A total of 14 journals now explicitly state on their websites that they publish replications. Interestingly, the net gain of 4 journals is the result of the addition of 6 journals that have newly stated they will publish replications, minus two journals that have removed mention of publishing replications: Economic Development and Cultural Change and the International Journal of Forecasting no longer explicitly state a policy of publishing replications.
Journals That Publish Replications
There are many journals that do not explicitly state they publish replications, but for which revealed preference shows they do. The left hand side of the table below reports published replications by journal for the time period covered by DPF&R. Note that the total of 206 published replications exceeds the number reported by DPJ&R. This is because I updated their list of replication studies and found additional studies that they missed. Through 2014, I identify a total of 21 journals that published more than 1 replication over their history. 5 journals were responsible for publishing approximately half of all replication studies, with the Journal of Applied Econometrics and the American Economic Review leading the pack.
Approximately three and a half years later, 26 journals have published more than 1 replication study. 5 journals still account for half of all published replication studies, with the same two journals leading the list. Notably, over half of the 71 replication studies published since 2014 have appeared in three journals: the Journal of Applied Econometrics, Econ Journal Watch, and Public Finance Review. The increase in Public Finance Review’s replications can be attributed to the fact that they introduced a dedicated replication section.
Journals That Require Authors to Provide Data and Code
In their article, DPJ&R surveyed the list of Web of Science “Economics” journals that “regularly” provided data and code for their empirical articles. “Regularly” was defined as providing data and code for at least half of the journal’s empirical articles in recent issues. They reported 27 articles met this criterion.
We updated the list of 360 Web of Science “Economics” journals and repeated the analysis. Only this time, we also kept track of which journals required authors to supply their data and code. A total of 41 journals explicitly stated that authors of empirical papers were required to provide data and code sufficient to replicate the results in the paper. Journals that said they “encouraged” authors to provide data and code, or that required authors to make their data and code available “upon request”, were not included in this list.
We then went through 15 of each journal’s most recently published empirical articles. If at least half (8) had both data and code, we classified it as satisfying the journal’s requirement. 23 of the 41 (56%) met this criterion. These are listed in the left hand side of the table below.
If less than half of the articles were accompanied by data and code, or if only data and not code were provided, or code and not data, we judged it to have not satisfied the criteria. In this manner, 18 of the 41 journals (44%) were determined to not be in compliance with their stated data and code policy. These are listed on the right hand side of the table.
Please note, however, an important caveat: most journals make an exception for confidential data. These could be data that were provided with strict confidentiality requirements, or subscription data where the vendor does not want to make the data public. We had no way of knowing if this was the reason an article did not provide data and code. Thus, some of the journals on the right hand side of the table may be in compliance with their data and code policy once one accounts for confidentiality restrictions.
Conclusion
Putting the above together, what can one make of the current status of replications in economics? Based upon what one sees in the journals, this is a case of water in the glass. For those who are optimists, the glass may be seen as half full. More journals are publishing more replication studies now than they were a decade ago. More journals are announcing that they publish replications. And more journals (one more!) are posting data and code along with their empirical articles. Replications and reproducibility are inching their way forward in the economics discipline.
However, for those who think there is a replication crisis in economics, the glass is half empty, and arguably not even that. The situation is a far cry from what is happening in some other social sciences, particularly psychology. There, articles like “Making Replication Mainstream” speak to a major culture change that seems to have been embraced by many, if not most, editors of leading journals in that discipline.
Some would argue that the reason psychology has been so willing to embrace replication is because that discipline has been more prone to questionable research practices. While that may be the case, the fact is, nobody really knows. There is only way to find out just how good, or bad, things are in economics. And that is to do more replications. Based on the above, it will be sometime yet before enough replications are done so that we can have a better idea of the status of reproducibility in the economics discipline.
Bob Reed is a professor of economics at the University of Canterbury in New Zealand. He is also co-organizer of the blogsite The Replication Network. He can be contacted at bob.reed@canterbury.ac.nz.
[From the article “The preregistration revolution” by Brian Nosek, Charles Ebersole, Alexander DeHaven, and David Mellor, published in PNAS]
“Progress in science relies in part on generating hypotheses with existing observations and testing hypotheses with new observations. This distinction between postdiction and prediction is appreciated conceptually but is not respected in practice. Mistaking generation of postdictions with testing of predictions reduces the credibility of research findings. … An effective solution is to define the research questions and analysis plan before observing the research outcomes—a process called preregistration.”
“…A substantial barrier to preregistration is insufficient or ineffective training of good statistical and methodological practices. Most researchers embrace the norms of science and aim to do the most rigorous work that they can (57). Those values are advanced with education and resources for effective preregistration in one’s research domain.”
“The reference list for this review provides a starting point, and there are some education modules available online to facilitate preregistration planning: for example, online courses (https://www.coursera.org/specializations/statistics, https://www.coursera.org/learn/statistical-inferences), instructional guides (help.osf.io/m/registrations/l/546603-enter-the-preregistration-challenge), criteria established for preregistration badge credentials (https://osf.io/6q7nm/), and collections of preregistration templates (https://osf.io/zab38/wiki/home/).”
“…Existing domain-specific and domain-general registries make it possible for researchers in any discipline to preregister their research. The World Health Organization maintains a list of registries by nation or region (http://www.who.int/ictrp/network/primary/en/), such as the largest existing registry, https://clinicaltrials.gov/. While focused on clinical trials in biomedicine, many of these registries offer flexibility to register other kinds of research.”
“The AEA RCT Registry, the American Economic Association’s registry for randomized controlled trials (https://www.socialscienceregistry.org), the Registry for International Development Impact Evaluations (RIDIE) Registry (ridie.3ieimpact.org/), and the Evidence in Governance and Politics (EGAP) Registry (egap.org/content/registration) are registries for economics and political science research. The Open Science Framework (OSF) (https://osf.io) is a domain-general registry service that provides multiple formats for preregistration (https://osf.io/registries/), including the flexible and relatively comprehensive Preregistration Challenge format (https://osf.io/prereg/).”
“Finally, the website https://aspredicted.org/ provides a simple form for preregistration, but it is not itself a registry because users can keep their completed forms private forever and selectively report preregistrations. However, researchers can post completed forms to a registry to meet the preservation and transparency standards.”
[From the article, “First analysis of ‘pre-registered’ studies shows sharp rise in null findings” by Matthew Warren, published at Nature.com]
“Studies that fail to find a positive result are often filed away, never to see the light of day, which leads to a publication bias that compromises the credibility of scientific literature.”
“An analysis now suggests that registering and peer-reviewing study protocols before research is conducted could improve this ‘file-drawer problem’, and help to correct the existing publication bias towards positive findings.”
“Researchers from Cardiff University, UK, report what they say is the first analysis of whether the practice is effective. They find that studies for which protocols were pre-registered are much more likely than the general scientific literature to report null findings. The analysis was posted on 17 October to the PsyArXiv repository.”
[From the website of Claremont McKenna College, USA]
“Claremont McKenna College’s Program on Empirical Legal Studies (PELS) is pleased to announce the second annual Empirical Legal Studies Replication Conference to be held on Friday, April 26, 2019, in Claremont, California.”
“…We are currently soliciting proposals that detail a plan to replicate an important study in law and economics/empirical legal studies. We welcome submissions using all methodologies and the conference will include a range of replication methodologies. We are particularly interested in papers that focus on natural language replication of previously hand-coded datasets or replications of randomized control trials. We also encourage proposals from teams consisting of a faculty member and one or more graduate students, in recognition of the fact that replication studies are also an excellent form of pedagogy.”
“…PELS will reimburse up to one graduate student and one faculty member for accommodations and up to $500 in travel costs. Please email inquiries and proposals to Eric Helland at ehelland@cmc.edu. “
In the past, the findings of numerous replication studies in economics have raised serious concerns regarding the credibility and reliability of published applied economic research. Literature suggests several explanations for these findings: Beyond missing incentives and rewards for the disclosure of data and code, only a small proportion of journals in economic sciences have implemented functional data availability policies. Also authors frequently do not comply with the demands of these policies or provide insufficient data and code. Our paper (“Journals in Economic Sciences: Paying Lip Service to Reproducible Research?”) regards an additional aspect and asks to which degree editorial offices enforce the data policies of their journals.
To date, only a minority of journals in economics possesses a policy on the disclosure of data and code that has been used to achieve the results of an empirical research article. But the count is growing over the years. Drivers of this shift could be located in the ongoing debates on replicable research and also in the growing demands of research funders and science policy.
In our paper we ask how much journals with a data policy enforced their policy in the past. To answer this question, we started our analyses with an evaluation of 599 articles published by 37 journals with a data availability policy. All articles have been published in the years 2013 and 2014.
At first, our analysis carved out the share of articles that fall under a data policy, because replication data is needed to verify the results of these articles. In total, we classified more than 75% of these articles to be empirical – or as we defined it, to be ‘data-based’.
Afterwards, we checked the journal data archives (if available) and the supplemental information section of each data-based article for the availability of replication files. We distinguished between articles using restricted data and such using non-restricted data. On average, only slightly more than a third of the data-based articles had accompanying data and code available.
Subsequently, we compared the demands of journals’ data policies with the replication files available for a subsample of 245 articles published by 17 journals in detail. Thereby, we were able to determine for each of the journals investigated how much the journal enforces its data policy.
For the years 2013 and 2014, our findings suggest a mixed picture: While one group of journals achieved high or very high compliance rates, a significant share of journals only sporadically provides replication files.
Due to the limited sample size and our focus on two years of publication, our analysis only provides a snapshot of journals’ practises at that time. Therefore, our findings should not be seen as a general overview of journals’ willingness to enforce their data policies. Also, our findings make not statement regarding the replicability of the research findings. We only checked for the availability of the prerequisites for potential replication attempts.
Based on the outcome of our analyses, we recommend editorial offices to pay more attention whether journal’s data policy has been fulfilled by their authors. Journals should be stricter in enforcing their data policies, because replicability of published research is a cornerstone of the scientific method.
In the first place editors are accountable for enforcing journals’ data policies, but also reviewers should feel a responsibility to take care of a periodical’s data policy. Both, editors and reviewers, invest time in ensuring that authors comply with journal’s style sheet. To also invest efforts in ensuring that replication files are available according to journal’s data policy is a task that would strengthen the scientific reputation of a periodical furthermore.
First and foremost journals play a crucial role for the scientific quality assurance. Thereby journals are also important for promoting a culture of research integrity because published articles are the most visible output of research.
In the spirit of replicable research, you can find the data, code and supplemental information of our analysis in the ZBW Journal Data Archive.
Sven Vlaeminck is a research assistant at the ZBW – Leibniz Information Centre for Economics. He also is the product manager of the ZBW Journal Data Archive. His biographical and contact information can be found here.
[From the “Call for a special issue of the Journal of Economics, Race, and Policy”]
“The Journal of Economics, Race, and Policy (JERP) has a commitment to publishing high-quality, replication research. Accordingly, we are making a call for a special issue of replication studies of important research of economic outcomes of both public and private sector market forces on groups distinguished by racial, tribal, ethnic, gender, and linguistic differences.”
“Many papers on race/gender/ethnicity/etc, over the years, have drawn a great deal of attention because they were ground breaking. However, many have also drawn scrutiny due to perceived poor analysis, questionable data, or outright fraud.”
“This special issue will be dedicated to Positive (or validating) replications, Negative (unable to reproduce) replications, and Negative (able to reproduce but find the results are not robust) replications.”
“We strongly urge academic economists, sociologists, political scientists, statisticians, and graduate students beginning their research careers to undertake replication experiments. This special issue will be an excellent opportunity.”
“If you would like to be the guest editor or contribute to the special issue … contact me [Gary Hoover, Editor-in-Chief] for more information.”
Information about the journal can be found here. The editor can be contacted at ghoover@ou.edu.
[From the paper, “Ensuring the quality and specificity of preregistrations” by C.L.S. Veldkamp et al., posted at PsyArXiv Preprints]
“…we evaluated two preregistration formats that are available on the OSF: a lightweight format that maximizes flexibility for the researcher to define preregistration content that is most fitting for their research (“Standard Pre-Data Collection”), and a highly-defined format that provides a specific workflow and instructions for what must be preregistered (“Prereg Challenge”). We evaluated these two preregistration formats for their independent and comparative effectiveness at reducing researcher degrees of freedom in design and analysis that could affect the credibility of statistical inferences.”
“…To evaluate the extent to which Standard Pre-Data Collection Registrations (SPR) and Prereg Challenge Registrations (PCR) restricted potential opportunistic use of Researcher Degrees of Freedom (DFs), we constructed a coding protocol based on 29 of the 34 DFs from Wicherts et al. (2016; Table 1)…”
“…As expected, PCR generally restricted opportunistic use of DFs better than SPR. In addition, we examined for each DF separately whether it was restricted better in PCR or in SPR. Twenty-two of 29 DFs were better restricted by PCR than by SPR, but only 14 of 29 were significantly so (p < .05). Moreover, the median effect size by DFs was 0.17 (corresponding to a small effect size) and only two showed large effects in the expected direction …”
“…our conclusion: Preregistrations will be more effective at restricting researcher degrees of freedom by using protocols with specific, comprehensive instructions about what needs to be reported.”
The Replication Network 
You must be logged in to post a comment.