[From the article “We’ve Been Deceived: Many Clinical Trial Results Are Never Published”] It is now common practice for clinical trials to register their protocols prior to enrolling participants. These efforts are important if the research community is to have a better understanding of the research that has been done, and is ongoing. Such efforts are also useful to constrain data mining and the discovery of spurious correlations.
However, just because a trial is registered, doesn’t mean that the results are reported. And this contributes to the phenomenon of publication bias. A new tracking system called “TrialsTracker” is intended to identify the extent to which the results from clinical trials are not reported. The article reports that Canadian research institutions are among the worst when it comes to not reporting the results of clinical trials. To read more, click here. Elsewhere, studies have shown that trials with negative results are twice as likely to go unreported than trials that produce statistically significant results (see here).
In a recent paper, Schweinsberg et al. (2015) propose the idea of a “pre-publication independent replication” (PPIR). The idea is that an author(s) with a one or more studies that have identified interesting results but that have not yet been published, coordinate with a number of other researchers to have their initial results replicated.
Designed for experimental results in social psychology, but not necessarily limited to those, the original authors solicit other laboratories after obtaining their results. They collaboratively work out the replication protocols and criteria. Ideally, multiple replicating labs are involved so as to increase sample sizes.
PPIR is unlike other replication efforts such as the Reproducibility Project and Many Labs Projects because (i) findings are replicated before they are published, and (ii) the author of the original studies selects labs to replicate the results so as to ensure replicator expertise. PPIR is a way to ensure that new results are vetted before they work their way into the literature and become accepted as “fact.”
The authors demonstrate their approach by replicating ten experimental findings on “moral judgement effects” produced by social psychologist Eric Uhlmann at INSEAD and coauthors, employing 25 laboratories as replicators. They obtained a replication rate of 60-80%. The paper also discusses various issues associated with PPIR and other replication approaches. To read more, click here.
[From the website, The Scholarly Kitchen] “Data availability and re-usability starts with best practices in collecting and storing data in the first place. The exasperatingly funny video below shows what happens when those best practices are ignored, something that’s much more prevalent than it should be.” To check out the video, click here.
PS. We await the sequel where the researcher finally is able to read the data and run the regressions, but then is not able to reproduce the results reported in the published article.
[From the blogsite, Data Colada] As noted previously in TRN, the Social Sciences Replication Project is replicating 21 experimental studies published in Nature and Science from 2010-2015. To determine whether the original studies replicate, the associated team of researchers is using the following rule: “Set n for the replication so that it would have 90% power to detect an effect that’s 75% as large as the original effect size estimate. If “it fails” (p>.05), try again powering for an effect 50% as big as original.”
URI SIMONSOHN argues that this “90-75-50” rule is “noisy and wasteful.” He contrasts it with his own “small telescopes” approach to replication and finds that his approach produces more reliable findings with a more efficient use of sample sizes. The linked “Small Telescopes” article provides an informative discussion of some of the issues involved with what it means “to replicate”. To read more, click here.
EDaWaX stands for European Data Watch Extended. It recently introduced a new service, the “ZBW Journal Data Archive”, to assist journals in storing and managing published economic research. This new service of the German National Library of Economics (ZBW) is free of charge to academic journals. Here is EDaWaX’s story.
AN INFRASTRUCTURAL NEED ARTICULATED BY THE ECONOMIC COMMUNITY…
The project European Data Watch Extended (EDaWaX) started in fall 2011, after editors of a scholarly journal in economics discussed the idea of developing a suitable infrastructure for replication files of economic journals jointly with ZBW. The partners received funding from the German Research Foundation (DFG).
For several decades, researchers such as the economist B.D. McCullough have highlighted the poor record of replicability in applied economic research. Out of this came a call for journals to implement mandatory data availability policies with corresponding data archives. However, a suitable infrastructure did not exist. A notable exception was Dataverse – developed by IQSS at Harvard University. Indeed, Dataverse, has evolved as a data depository for both individuals and journals (such as Economics E-Journal and Review of Economics and Statistics).
…AND A NEW FIELD FOR RESEARCH LIBRARIES
For ZBW, EDaWaX was one of its first projects in the field of research data management. Together with partners from the German Data Forum, the Max Planck Institute for Innovation and Competition and – at a later stage – the Research Data Centre of the Socio-Economic Panel (FDZ-SOEP), EDaWaX started by evaluating the existing state of affairs. This analysis included examinations of journal data policies, data sharing behaviour of economists and surveys on perceptions of editorial offices towards data availability policies. EDaWaX then checked if services for journals to manage replication data were available among scientific infrastructure service providers in Germany and elsewhere.
From these beginnings, the project created a web-based application and developed a metadata scheme. The application uses CKAN – an open source software broadly used by open data portals of public sector entities around the globe. At the end of EDaWaX’s first funding term, a pilot application was available and, in late autumn 2013, the project presented its solution to a gathering of journal editors in the social sciences.
FROM PILOT APPLICATION TOWARDS A SERVICE
In the second funding period, between 2014 and 2016, the project worked on enhancing the application and making it fit for service. A key focus was developing the capability of the metadata component. Dealing with metadata is a balancing act: On the one hand, one would like to have as much metadata for the replication files as possible, because well described data and files are much more likely to be reused. In addition, the metadata can be used to discover the replication files in disciplinary portals, search engines and so on. But on the other hand, researchers are not willing to invest much of their time in creating metadata. Therefore, the project spent much effort to lower the burden of creating metadata. The key was automatisation whenever possible. Another feature was making the metadata field adaptable for different resources (e.g. datasets need more additional context information, while program code or documentation need less).
In addition, EDaWaX also implemented the possibility to mint DOIs (Digital Object Identifiers) for replication files. The main idea behind using a Persistent Identifier is to credit researchers for their investment of time and for sharing their data: The data can be cited appropriately (just like a traditional publication) and therefore authors gain an incentive to share and to document their data.
Finally, an important task was to present the web service to the community. The project held several workshops at annual meetings of German learned societies in economics and management, but also presented its work at the 2016 annual meeting of the American Economic Association (ASSA). In total the project gave more than 27 talks and workshops at national and international conferences.
WORKING WITH THE ZBW JOURNAL DATA ARCHIVE
Working with the web service is easy and time-saving for editorial offices: In a nutshell, editorial offices have to register their authors to the ZBW Journal Data Archive. The authors subsequently receive an email and create their personal accounts. Afterwards, authors may deposit the replication files of their articles at the journal data archive. In addition, they are asked to describe their files with metadata. For assistance, manuals in English and German are available.
When an author has completed the deposit of his/her replication files, the editorial office receives a notification by the web service. In a next step, the editorial office approves the replication files and checks the metadata for consistency. The editors supplement the URL or DOI of the published article, the page numbers, issue and volume of the journal. Subsequently, the replication files are ready to be published and to receive a DOI.
Currently, the Journal of Economics and Statistics (listed in the ‘Social Sciences Citation Index’) is utilising the ZBW Journal Data Archive. Other journals will follow in the next months.
Journals and editorial offices interested in employing the ZBW Journal Data Archive are warmly invited to contact the author to learn more about the software and how it can be of service to your journal.
Sven Vlaeminck is Project Manager for Research Data at ZBW – German National Library of Economics and the Leibniz Information Centre for Economics. He can be contacted at s.vlaeminck@zbw.eu.
[From the website of The Poverty Action Lab, Department of Economics at MIT] “The Abdul Latif Jameel Poverty Action Lab (J-PAL) at MIT’s economics department announces a fellowship program offering financial support (tuition assistance of up to $12,000 and a stipend of $13,000) for one semester (approximately 4.5 months) for current PhD students. While preference will be given to students from economics programs, graduate students from other disciplines with strong quantitative and programming skills (STATA, R etc.) are very welcome to apply. During the fellowship, students will work with some of J-PAL’s 150 affiliated professors from dozens of universities around the world, re-analyzing RCTs from scratch (starting with the creation of the data set from “raw survey” data, up to production of the final econometric analysis included in the working papers).” To learn more, click here.
The Open Science Framework announces the opening of a prediction market (PM) to accompany a new replication project. Over the period September 2016 to September 2017, 21 experimental studies that were published from 2010-2014 in the journals Science and Nature will be replicated. For each study, PM participants will be able to bet on the following proposition: “The effect in the replication is in the same direction as in the original study, and is statistically significant with a p-value smaller than 0.05.”
There will be two chances to replicate each study. The first study will select a sample size corresponding to 90% power to detect 75% of the effect size reported in the original study. If the first replication attempt is unsuccessful, a second study will be conducted with a sample size corresponding to 90% power to detect 50% of the effect size reported in the original study.
To learn further information about the 21 studies being replicated, the protocols being used used to replicate each study, the prediction markets, and other details associated with the replication project, click here.
[From the website of the Association for Psychological Science] “A coordinated replication effort conducted across 17 labs found no evidence that surreptitiously inducing people to smile or frown affects their emotional state. The findings of the replication project are published as part of a Registered Replication Report (RRR) in Perspectives on Psychological Science, a journal of the Association for Psychological Science.” To read more, click here.
Professor Eric-Jan “EJ” Wagenmakers, Professor of Psychology at the University of Amsterdam, has been a leading advocate for pre-registration, replication, and the use of Bayesian statistics, particularly in replication studies. An interview that highlights his professional contributions can be found here.
Readers interested in replications should check out his course, “Good Science, Bad Science”. While focused on psychological science, the course should be of interest to many social scientists, including economists.
Over 16 classes, students are walked towards the following goals:
— “To understand the current “crisis of confidence” in psychological science, and learn what can be done to fix it.”
— “To obtain guidance and practical experience in designing informative and honest replication experiments through preregistration.”
— “To submit four high-quality preregistration proposals to peer-reviewed international journals for ‘in principle acceptance'”
To learn more about the course, click here.
[This blog originally appeared at the blogsite Development Impact] About a year ago, I blogged on a paper that had tried to replicate results on 61 papers in economics and found that in 51% of the cases, they couldn’t get the same result. In the meantime, someone brought to my attention a paper that takes a wider sample and also makes us think about what “replication” is, so I thought it would be worth looking at those results.
The paper in question is by Maren Duvendack, Richard Palmer-Jones and Robert Reed and appeared last year in Econ Journal Watch (see here). The paper starts with an interesting history of replication in economics. It turns out that replication goes pretty far back. Duvendack and co. cite the introductory editorial to Econometrica, where Frisch wrote “In statistical and other numerical work presented in Econometrica the original raw data will, as a rule, be published, unless their volume is excessive. This is important to stimulate criticism, control and further studies.” That was in 1933.
Various journals have made similar affirmations of the need for replication over the years. The Journal of Human Resources put it in its policy statement in 1990 – explicitly saying that it welcomed the submission of studies that replicated studies that had appeared in the JHR in the last five years. But this is missing from the current policy, which focuses more on making data and code available with published papers. The Journal of Political Economy took a different approach, and had a “confirmations and contradictions” section from 1976-1999. These explicit publication opportunities may have declined in recent times, but there has been a sharp surge in a different path to replication – the requirement that authors submit their code and dataset for a given paper. Duvendack and co. find 27 journals that regularly publish data and code – and many of these are top journals. The only development field journal that makes this list is the World Bank Economic Review. In addition, many funders now require that, after a decent interval, the data they funded be made publicly available in its entirety.
Before we look at Duvendack and co.’s review of replication trends, it’s worth taking a short detour as to what exactly replication means. Unfortunately, as it’s used in many conversations, it’s imprecise. Michael Clemens has a very nice (and very precise) paper where he lays out a number of distinctions. In this case, precision requires some verbosity, so hang on. Clemens lays out four different types (in two groups):
Replication (both sub-types use the same sampling distribution of parameter estimates and are looking for discrepancies that come from random chance, error, or fraud):
– Verification – uses the same specification, same population and same sample
– Reproduction – uses the same specification, same population but not the same sample
Robustness (uses different sampling distribution for parameter estimates and is looking for discrepancies that come from changes in the sampling distribution – as Clemens notes they need not give identical results in expectation):
– Reanalysis – uses a different specification, the same population and not necessarily the same sample
– Extension – uses the same specification, different population and a different sample.
Duvendack and co. are using a broader definition of replication (especially when compared to the paper I blogged on last year): they’re including what Clemens calls robustness. They go out, casting a wide net to look for replication studies (they include not only Google Scholar and the Web of Science, and the Replication in Economics wiki, but also suggestions from journal editors, their own collections and a systematic search of the top 50 economics journals). This search gives them 162 published studies. The time trend is interesting, as the figure below (reproduced from their paper) shows what could be an upwards trend:
One development journal that contributed significantly to these 162 studies was the Journal of Development Studies. We can’t tell how many other development papers are in the 162 since the rest of the main contributors are more general interest journals. The Journal of Applied Econometrics (JAE) is the main overall contributor to this body of work – they clock in with 31 replications – in large part because they have a dedicated replication section which can consist of pretty short summaries.
Duvendack and co. then look at the characteristics of these replications. I am going to focus on the non-JAE, non-experimental (as in experimental economics, not field experiments) studies which number 119. About 61 percent of these studies are an “exact” replication or, to use Clemens’ taxonomy, a verification study. 55 percent of studies extend the original findings. As might be expected, the majority of the 119 studies (73 percent) find a significant difference with the original result. About 17 percent confirm the previous study and 10 percent are mixed. And 26 percent of studies have a reply by the original study authors.
As Duvendack and co. point out, we shouldn’t think of the published studies as a random sample. Which brings us to incentives. Clearly getting confirmatory studies published in major journals is going to be hard, and particularly hard for simple verification studies. Turning to results that disagree with the original study, Duvendack and co. speculate that some journals might be reluctant to publish contradictions of influential authors. I can also imagine that younger researchers may be averse to taking on this particular challenge, given that influential senior researchers may show up in their career futures. Returning to the journal side of the equation, it also doesn’t look particularly good for the journal to take down one of their own papers. On another level, for journals the citation per page count is likely to be significantly lower for a replication than for an original paper (although Duvendack and co. suggest this could be alleviated by publishing very short replication papers with the longer paper as an online appendix). Finally, the incentives for authors of the original study to make replication easy are pretty weak – if someone confirms your study it’s not really a big deal, but it’s a big deal if they don’t.
So all of these factors point towards the lower likelihood of replications. There are a couple of factors that might make replications more likely. The first is there are somewhat more communities to support this than there used to be. Beyond the wiki mentioned above, Duvendack and co. have a list inside economics (e.g. 3ie’s work), but also for other disciplines. In addition, the growth in online storage, the growth in computer processing power, plus the increasing number of journals requiring the posting of code and data lower the costs of replication dramatically. And the trend is positive, so maybe there is some social support. It will be interesting to see what the future brings.
Markus Goldstein is a development economist with experience working in Sub-Saharan Africa, East Asia, and South Asia. He is currently a Lead Economist in the Office of the Chief Economist for Africa at the World Bank.
The Replication Network 
You must be logged in to post a comment.