[From the article “The changing forms and expectations of peer review” by Serge Horbach and Willem Halffman, published in Research Integrity and Peer Review, 2018, 3:8]

This is a wonderful article that provides a comprehensive discussion of peer review in the context of scientific quality and integrity. Here are some highlights from the article.

– Provides context for arguments around the role of peer review in ensuring scientific quality/integrity. This includes references from those arguing that it performs that function adequately, to others that argue it fails miserably.

– It discusses the historical evolution of peer-review, arguing that it did not become a mainstream journal practice until after the Second World War.

– Explains how the desire to ensure fairness and objectivity led to single-blind, double-blind, and triple-blind reviewing (where even the handling editor does not know the identify of the author). See table below:

– Discusses the evidence for bias (particularly gender and institutional-affiliation bias) in peer review.

– It is interesting that the same concern for reviewer bias has led to diametrically opposite forms of peer review: double-blind peer review and open peer review.

– With the advent of extra-journal publication outlets, such as pre-print archives, there has been discussion that peer review should serve less the role of quality assurance, and more the goal of providing context and connection to existing literature.

– Makes the argument that one of the motivations behind “registered reports”, where journals decide to publish a paper based on its research design — independently of its results — is that this would provide a greater incentive to undertake replications.

– Related to the replication crisis and publication bias, peer review at some journals has moved to re-focussing assessment away from novelty and statistical significance, and towards importance of the research question and soundness of research design.

– Another development in peer review has been the creation of software to assist journals and reviewers in identifying plagiarism and to detect statistical errors and irregularities.

– Artificial intelligence is being looked to in order to address the burdensome task of reviewing ever-increasing numbers of scientific manuscripts. The following quote offers an intriguing look at a possible, AI future of peer review: “Chedwich deVoss, the director of StatReviewer, even claims: ‘In the not-too-distant future, these budding technologies will blossom into extremely powerful tools that will make many of the things we struggle with today seem trivial. In the future, software will be able to complete subject-oriented review of manuscripts. […] this would enable a fully automated publishing process – including the decision to publish.’”

– Given the increasingly important role that statistics play in scientific research, there is an incipient movement for journals to employ statistical experts to review manuscripts, including the contracting of reviewing to commercial providers.

– Post-publication review, such as that offered by PubPeer, has also expanded peer review outside the decision to publish research.

– Another movement in peer review has been to introduce interactive discussion between the reviewer, the author, and external “peers” before the editor makes their decision. Though this is not mentioned in the article, this is the model of peer review in place at the journal Economics: The Open Access, Open Assessment E-journal.

– The article concludes the discussion by noting that as academic publishing has become big business, with high submission and subscription fees charged to authors and readers, there is an increasing sense that academic publishers should be held responsible for the quality of their product. This has — and will have even more so in the future — consequences for peer review.

To read the full article, click here.