(FROM THE ARTICLE “Psychology’s Replication Crisis Has a Silver Lining”) The author, a prominent professor of psychology at Yale University, argues that not all is gloom and doom. Some failures to replicate are due to noise swamping signal. Other failures to replicate are due to faulty replicators, not faulty original research. And much research IS replicable. Finally, as bad as things are in psychology, at least the situation is better than in medical research. To read more, click here.
[From the website The Edge] The question is: “What scientific concept would improve everybody’s cognitive toolkit?” Stanford Associate Professor BRIAN KNUTSON’s answer is: “Replication.” He suggests that a more meaningful measure of positive scientific impact is a Replication Index, or “r-factor.” This would measure the extent to which a researcher’s results could be independently confirmed. To boost their “r-factors”, researchers would be incentivized to make their work more transparent, making it easier for others to confirm their results. To read more, click here.
Replications are hot. Don’t believe me, just read The Economist. This is the third story related to scientific reliability/replications this past month. To read more, click here.
(FROM THE CHANNEL’S WEBSITE) “The Preclinical Reproducibility and Robustness channel is a platform for open and transparent publication of confirmatory and non-confirmatory studies in biomedical research. The channel is open to all scientists from both academia and industry and provides a centralized space for researchers to start an open dialogue, thereby helping to improve the reproducibility of studies.” To read more, click here.
(FROM THE ARTICLE “Are Results in Top Journals To Be Trusted?”) A paper recently published in the American Economic Journal, entitled “Star Wars: The Empirics Strike Back”, “analyses 50,000 tests published between 2005 and 2011 in three top American journals. It finds that the distribution of results (as measured by z-score, a measure of how far away a result is from the expected mean) has a funny double-humped shape (see chart below). The dip between the humps represents “missing” results, which just happen to be in a range just outside the standard cut-off point for statistical significance (where significance is normally denoted with stars, though the name may also be something to do with a film recently released—file under ‘economists trying to be funny’). Their results suggest that among the results that are only just significant, 10-20% have been fudged.” To read more, click here.
As reported in a previous blog post, the Economics E-Journal has launched a new replication section. As part of this initiative, we have developed a set of guidelines for replication submissions.
These guidelines seek to strike a reasonable balance among the needs of replicating authors (a fair chance to publish replications), replicated authors (protection against poorly-done replication studies), and readers (who need to know in a timely manner whether or not economics research is robust).
These guidelines for replication submissions are described at the journal’s website. This blog consists of two parts. In the first part, we provide a summary of our current guidelines. In the second part, we ask readers for input.
PART I: Current Guidelines
The guidelines for a replication submission at Economics E-Journal are summarized as follows.
1) An assistant editor determines whether the submission is of sufficient merit to be sent through the refereeing process. If not, the paper is desk-rejected.
2) If a paper passes the first stage, it is sent on to a Co-Editor, who makes a similar determination about merit. If the paper is not of sufficient merit, the paper is desk-rejected.
3) Then, the replication is sent to the original author, who has a chance to reply within 60 days. This reply is then appended to the submission.
4) If the paper passes this stage, the paper (with reply) is published as a discussion paper (which is like a working paper).
5) It is then sent on to two or three anonymous referees, none of whom is the original author. These referees submit reports which are posted online. Commenters can also comment during this time.
6) After the referee reports are posted, the author may reply to the referees or even update the paper.
7) After the author replies, a committee of three then makes a decision (generally to publish as a full-fledged journal article with or without specific revisions, or to reject). Any further exchanges between the replicating and original authors are then appended to the published article.
Our complete guidelines for replicators can be found here. While the guidelines are mostly set, we are still seeking input into our procedures, and we will undoubtedly make changes as we gain more experience with replications.
PART II: How You Can Help
We are seeking input on the following items:
– In light of the guidelines for non-replication submissions, do you believe that the current guidelines for replication submissions are appropriate?
– If not, what concrete suggestions do you have for improvement?
– Do you believe that the 60-day embargo on the discussion paper (to wait for the original author’s reply) makes sense, or should the discussion paper be published as soon as a Co-Editor believes it has sufficient merit?
– Should an embargo be placed instead on publication of the journal article? That is, should publication as a journal article wait until the original author has a chance to reply to the final version of the replication study?
In addition, we are keen to hear any other ideas you have for improving the replication policy at Economics E-Journal.
To provide feedback, comment directly on this blog page, or email Claire Boeing-Reicher (Kiel Institute for the World Economy), at Claire.Reicher@ifw-kiel.de .
We look forward to hearing from you.
In a blog for Retraction Watch, LIZ WAGNER argues that it is good when authors provide data and code. But it’s not necessarily the most important thing. Registering a research protocol would do more to prevent data mining and p-hacking. And requiring the former may distract attention from the latter. To read more, click here.
The Economics E-Journal announces the launch of a dedicated replication section. This initiative is a joint effort of the Kiel Institute for the World Economy (IfW) and the German National Library for Economics (ZBW). It provides authors across all fields of economics an outlet for publishing replication studies.
This initiative is motivated by the difficulties that authors have had in submitting replication studies to other journals, and by the culture of secrecy within the profession around failed replications. We hope that our initiative can help begin to change these things.
The journal has several unique characteristics that may make it an attractive outlet for researchers looking for outlets for their replication studies. These characteristics are driven by a principle of openness.
This openness shows up in the desire for the journal to open up access to the submission and refereeing processes, speed up those processes while maintaining high standards, and open up access to articles for readers outside major academic institutions.
So far, this openness has met with success.
For instance, the journal currently has an impact factor of 0.644 (JCR Social Sciences Edition 2014), which places it between the Southern Economic Journal (IF = 0.683) and Applied Economics (IF = 0.613). Further, Economics E-Journal’s impact factor is likely to be biased downward since the journal is not yet ten years old.
The characteristics of the Economics E-Journal that make it an ideal outlet for replications are as follows.
First of all, the journal’s electronic format implies that there are no space constraints. The only constraint on the number of replications that can be published is the quantity and quality of the submitted replication studies.
Secondly, the management of the journal occurs alongside, but independently from, the IfW’s and ZBW’s other journals, and from other journals in the economics profession. This ensures a degree of independence not found in some other journals.
Thirdly, the journal is open access and open evaluation. Open access means that authors can reach a wide audience without running into a paywall. Open evaluation means that reviewers, commenters, and editors adjudicate papers in a fair, transparent, and rapid way.
Fourthly, the journal is a general interest journal, which means that it accepts submissions from all subfields of economics.
Guidelines for replicators can be found here. While the guidelines are mostly set, we are still seeking input into our procedures. In my next installment, I will be asking TRN readers for their comments and suggestions. Stay tuned!
-Claire Boeing-Reicher, Researcher, Kiel Institute for the World Economy.
(FROM THE ARTICLE “All at sea: Ideological divisions in economics undermine its value to the public“): “Sifting out the guff requires transparency, argued John Cochrane of the University of Chicago in another recent blog post. Too many academics keep their data and calculations secret, he reckoned, and too few journals make space for papers that seek to replicate earlier results. Economists can squabble all they like. But the profession is of little use to anyone if it cannot then work out which side has the better of the argument.“
A new initiative calls for journal reviewers to ask editors to request authors to provide data and supporting code/documentation before they agree to review the manuscript. From The Peer Reviewers Openness Initiative: “We suggest that … reviewers make open practices a pre-condition for more comprehensive review. This is already in reviewers’ power; to drive the change, all that is needed is for reviewers to collectively agree that the time for change has come.” To learn more about the initiative, click here. To read the supporting paper, click here. (H/T to Political Science Replication)
You must be logged in to post a comment.