“Weekend Reads”, the weekly summary by IVAN ORANSKY of Retraction Watch, recently listed two articles on Peer Review.  One, a blog by George Borjas, concerns the recent imbroglio at the American Economic Review involving an editor who oversaw the review of an article by one of her coauthors (read here).  The other, a comment in Nature entitled “Let’s make peer review scientific” (read here) reviews 30 years of progress, and lack of progress, in peer reviewing.  Both articles underscore the obvious to anybody who has even minimal experience with the reviewing process — peer review is a flawed process.  

What does this have to do with replications?  The real problem with peer review is thinking that it is the final arbiter of a paper’s value.  Peer review is but one step in a lengthy process.  It follows the circulation of a working paper for comments and the presentation of one’s research at seminars and conferences.  But the publication of a paper should not be the final stage in a paper’s review.  

If a paper is important and makes a valuable contribution, that research should be examined further.  Were the data handled correctly?  Would alternative formulations of the research question have given similar results? Were the results robust to reasonable perturbations in experimental design?  These are things that are difficult for reviewers to address, because they generally do not have access to a researcher’s data and code.

Even when a journal requires data and code to be made available, rarely do reviewers have access to these when they are doing their review.  They are only available after a paper has been accepted for publication.  And it is only after researchers have been able to go through a paper’s data and code that they can judge for themselves whether the paper’s conclusions are fragile or robust.

The problem with peer review is not so much a problem with peer review.  The problem with peer review is the scientific community’s elevation of peer review in the review process.  Peer review should be thought of as an intermediate stage in the review of a paper.  As one part of the gauntlet that a paper needs to run to establish its scientific worth.  Until it becomes the norm for authors to provide their data and code when submitting their research to journals, it will inevitably be the case that the real “review” will have to be done in the post publication phase of a paper’s life.  Through replication.