On September 29, a public debate was held on the campus of NYU. The subject of the debate was “Do Replication Projects Cast Doubt On Many Published Studies in Psychology?” The debate pitted Brian Nosek, director of the Center for Open Science and a leading advocate of the importance of replication, against Jason Mitchell, a professor in the Department of Psychology at Harvard. To read a summary of the debate, click here.
[From the article “Why Fake Data When You Can Fake a Scientist?”] “Hoss Cartwright, a former editor of the International Journal of Agricultural Innovations and Research, had a good excuse for missing the 5th World Congress on Virology last year: He doesn’t exist. Burkhard Morgenstern, a professor of bioinformatics at the University of Gottingen, dreamt him up, and built a nice little scientific career for him. He wrote Cartwright a Curriculum Vitae, describing his doctorate in Studies of Dunnowhat, his rigorous postdoctoral work at Some Shitty Place in the Middle of Nowhere, and his experience as Senior Cattle Manager at the Ponderosa Institute for Bovine Research. Cartwright never published a single research paper, but he was appointed to the editorial boards of five journals.” To read more, click here.
Fake Authors. Fake Data. Fake Journals. Fake Reviewers. Fake Citations. One could get discouraged. Maybe this is a good time to note that replications could help.
[From the article “We’ve Been Deceived: Many Clinical Trial Results Are Never Published”] It is now common practice for clinical trials to register their protocols prior to enrolling participants. These efforts are important if the research community is to have a better understanding of the research that has been done, and is ongoing. Such efforts are also useful to constrain data mining and the discovery of spurious correlations.
However, just because a trial is registered, doesn’t mean that the results are reported. And this contributes to the phenomenon of publication bias. A new tracking system called “TrialsTracker” is intended to identify the extent to which the results from clinical trials are not reported. The article reports that Canadian research institutions are among the worst when it comes to not reporting the results of clinical trials. To read more, click here. Elsewhere, studies have shown that trials with negative results are twice as likely to go unreported than trials that produce statistically significant results (see here).
In a recent paper, Schweinsberg et al. (2015) propose the idea of a “pre-publication independent replication” (PPIR). The idea is that an author(s) with a one or more studies that have identified interesting results but that have not yet been published, coordinate with a number of other researchers to have their initial results replicated.
Designed for experimental results in social psychology, but not necessarily limited to those, the original authors solicit other laboratories after obtaining their results. They collaboratively work out the replication protocols and criteria. Ideally, multiple replicating labs are involved so as to increase sample sizes.
PPIR is unlike other replication efforts such as the Reproducibility Project and Many Labs Projects because (i) findings are replicated before they are published, and (ii) the author of the original studies selects labs to replicate the results so as to ensure replicator expertise. PPIR is a way to ensure that new results are vetted before they work their way into the literature and become accepted as “fact.”
The authors demonstrate their approach by replicating ten experimental findings on “moral judgement effects” produced by social psychologist Eric Uhlmann at INSEAD and coauthors, employing 25 laboratories as replicators. They obtained a replication rate of 60-80%. The paper also discusses various issues associated with PPIR and other replication approaches. To read more, click here.
[From the website, The Scholarly Kitchen] “Data availability and re-usability starts with best practices in collecting and storing data in the first place. The exasperatingly funny video below shows what happens when those best practices are ignored, something that’s much more prevalent than it should be.” To check out the video, click here.
PS. We await the sequel where the researcher finally is able to read the data and run the regressions, but then is not able to reproduce the results reported in the published article.
[From the blogsite, Data Colada] As noted previously in TRN, the Social Sciences Replication Project is replicating 21 experimental studies published in Nature and Science from 2010-2015. To determine whether the original studies replicate, the associated team of researchers is using the following rule: “Set n for the replication so that it would have 90% power to detect an effect that’s 75% as large as the original effect size estimate. If “it fails” (p>.05), try again powering for an effect 50% as big as original.”
URI SIMONSOHN argues that this “90-75-50” rule is “noisy and wasteful.” He contrasts it with his own “small telescopes” approach to replication and finds that his approach produces more reliable findings with a more efficient use of sample sizes. The linked “Small Telescopes” article provides an informative discussion of some of the issues involved with what it means “to replicate”. To read more, click here.
EDaWaX stands for European Data Watch Extended. It recently introduced a new service, the “ZBW Journal Data Archive”, to assist journals in storing and managing published economic research. This new service of the German National Library of Economics (ZBW) is free of charge to academic journals. Here is EDaWaX’s story.
AN INFRASTRUCTURAL NEED ARTICULATED BY THE ECONOMIC COMMUNITY…
The project European Data Watch Extended (EDaWaX) started in fall 2011, after editors of a scholarly journal in economics discussed the idea of developing a suitable infrastructure for replication files of economic journals jointly with ZBW. The partners received funding from the German Research Foundation (DFG).
For several decades, researchers such as the economist B.D. McCullough have highlighted the poor record of replicability in applied economic research. Out of this came a call for journals to implement mandatory data availability policies with corresponding data archives. However, a suitable infrastructure did not exist. A notable exception was Dataverse – developed by IQSS at Harvard University. Indeed, Dataverse, has evolved as a data depository for both individuals and journals (such as Economics E-Journal and Review of Economics and Statistics).
…AND A NEW FIELD FOR RESEARCH LIBRARIES
For ZBW, EDaWaX was one of its first projects in the field of research data management. Together with partners from the German Data Forum, the Max Planck Institute for Innovation and Competition and – at a later stage – the Research Data Centre of the Socio-Economic Panel (FDZ-SOEP), EDaWaX started by evaluating the existing state of affairs. This analysis included examinations of journal data policies, data sharing behaviour of economists and surveys on perceptions of editorial offices towards data availability policies. EDaWaX then checked if services for journals to manage replication data were available among scientific infrastructure service providers in Germany and elsewhere.
From these beginnings, the project created a web-based application and developed a metadata scheme. The application uses CKAN – an open source software broadly used by open data portals of public sector entities around the globe. At the end of EDaWaX’s first funding term, a pilot application was available and, in late autumn 2013, the project presented its solution to a gathering of journal editors in the social sciences.
FROM PILOT APPLICATION TOWARDS A SERVICE
In the second funding period, between 2014 and 2016, the project worked on enhancing the application and making it fit for service. A key focus was developing the capability of the metadata component. Dealing with metadata is a balancing act: On the one hand, one would like to have as much metadata for the replication files as possible, because well described data and files are much more likely to be reused. In addition, the metadata can be used to discover the replication files in disciplinary portals, search engines and so on. But on the other hand, researchers are not willing to invest much of their time in creating metadata. Therefore, the project spent much effort to lower the burden of creating metadata. The key was automatisation whenever possible. Another feature was making the metadata field adaptable for different resources (e.g. datasets need more additional context information, while program code or documentation need less).
In addition, EDaWaX also implemented the possibility to mint DOIs (Digital Object Identifiers) for replication files. The main idea behind using a Persistent Identifier is to credit researchers for their investment of time and for sharing their data: The data can be cited appropriately (just like a traditional publication) and therefore authors gain an incentive to share and to document their data.
Finally, an important task was to present the web service to the community. The project held several workshops at annual meetings of German learned societies in economics and management, but also presented its work at the 2016 annual meeting of the American Economic Association (ASSA). In total the project gave more than 27 talks and workshops at national and international conferences.
WORKING WITH THE ZBW JOURNAL DATA ARCHIVE
Working with the web service is easy and time-saving for editorial offices: In a nutshell, editorial offices have to register their authors to the ZBW Journal Data Archive. The authors subsequently receive an email and create their personal accounts. Afterwards, authors may deposit the replication files of their articles at the journal data archive. In addition, they are asked to describe their files with metadata. For assistance, manuals in English and German are available.
When an author has completed the deposit of his/her replication files, the editorial office receives a notification by the web service. In a next step, the editorial office approves the replication files and checks the metadata for consistency. The editors supplement the URL or DOI of the published article, the page numbers, issue and volume of the journal. Subsequently, the replication files are ready to be published and to receive a DOI.
Currently, the Journal of Economics and Statistics (listed in the ‘Social Sciences Citation Index’) is utilising the ZBW Journal Data Archive. Other journals will follow in the next months.
Journals and editorial offices interested in employing the ZBW Journal Data Archive are warmly invited to contact the author to learn more about the software and how it can be of service to your journal.
Sven Vlaeminck is Project Manager for Research Data at ZBW – German National Library of Economics and the Leibniz Information Centre for Economics. He can be contacted at s.vlaeminck@zbw.eu.
[From the website of The Poverty Action Lab, Department of Economics at MIT] “The Abdul Latif Jameel Poverty Action Lab (J-PAL) at MIT’s economics department announces a fellowship program offering financial support (tuition assistance of up to $12,000 and a stipend of $13,000) for one semester (approximately 4.5 months) for current PhD students. While preference will be given to students from economics programs, graduate students from other disciplines with strong quantitative and programming skills (STATA, R etc.) are very welcome to apply. During the fellowship, students will work with some of J-PAL’s 150 affiliated professors from dozens of universities around the world, re-analyzing RCTs from scratch (starting with the creation of the data set from “raw survey” data, up to production of the final econometric analysis included in the working papers).” To learn more, click here.
The Open Science Framework announces the opening of a prediction market (PM) to accompany a new replication project. Over the period September 2016 to September 2017, 21 experimental studies that were published from 2010-2014 in the journals Science and Nature will be replicated. For each study, PM participants will be able to bet on the following proposition: “The effect in the replication is in the same direction as in the original study, and is statistically significant with a p-value smaller than 0.05.”
There will be two chances to replicate each study. The first study will select a sample size corresponding to 90% power to detect 75% of the effect size reported in the original study. If the first replication attempt is unsuccessful, a second study will be conducted with a sample size corresponding to 90% power to detect 50% of the effect size reported in the original study.
To learn further information about the 21 studies being replicated, the protocols being used used to replicate each study, the prediction markets, and other details associated with the replication project, click here.
[From the website of the Association for Psychological Science] “A coordinated replication effort conducted across 17 labs found no evidence that surreptitiously inducing people to smile or frown affects their emotional state. The findings of the replication project are published as part of a Registered Replication Report (RRR) in Perspectives on Psychological Science, a journal of the Association for Psychological Science.” To read more, click here.
You must be logged in to post a comment.