Benefits and Drawbacks of Open Peer Review | Lex Academic Blog

Concerns about implicit bias in traditional methods of peer review, including double-blind, have prompted some journal publishers such as PLOS and Frontiers to implement an open peer review policy. Advocates of open peer review argue that it encourages accountability, transparency and constructive criticism, while opponents contend that these supposed advantages in fact lead to reduced scholarly rigour. These arguments are difficult to evaluate because what is meant by open peer review varies from one journal to another and within the academic literature on the topic. Tony Ross-Hellauer’s systematic review of 122 definitions identifies seven characteristics, the most common of which are open identities (i.e. the identities of the author and reviewer are known), open reports (i.e. reviewer reports are published alongside accepted articles) and open participation (i.e. both non-specialists and invited experts can comment). These three traits will be the focus of this discussion of the benefits and drawbacks of open peer review.

These traits are more or less likely to feature in definitions of open peer review depending on the disciplinary focus of a study. Almost 94% of STEM-focused studies, which make up the majority of the literature on open peer review, consider open identities to be a characteristic of open peer review, compared to 84% of studies with a humanities and social sciences focus. Over 60% of STEM studies regard open reports as a trait of open peer review, compared to fewer than 40% of SSH studies. On the other hand, almost 54% of SSH studies include open participation in their definition of open peer review, more than double the proportion of STEM studies. As Ross-Hellauer points out, it would be hasty to draw conclusions from these statistics, given the relatively small number of SSH-focused studies and the interdisciplinarity of journal content. They are nonetheless interesting and warrant further research.

Open peer review unquestionably encourages constructive criticism. For the reviewer, simply knowing the author’s name is likely to create a human connection that pushes them to identify strengths, as well as weaknesses, in a piece of work, which will regulate the tone of feedback. Open identities render irrelevant some of the flaws in double-blind peer review, such as self-citation and the possibility of identifying a supposedly anonymous author. From the author’s perspective, it is nice, for want of a better word, to know the name of the reviewer. Publishing reviewer reports alongside articles makes reviewers accountable, thus further encouraging constructive criticism, because their reviews will themselves be reviewed by readers. Combined, open identities and open reports pave the way for exchange and debate, and the opportunities for extending scholarship these bring.

Open peer review, and open reports in particular, help to extend scholarship because they not only foster relationships but also enrich the academic record. When reviews are published, readers can see the limitations of a piece of work at review stage and how they were addressed, and thus take their own critical stance on these limitations. In addition, open reports serve as models of good practice for researchers new to peer reviewing, guiding them on content, length and tone. Finally, they are a recognition of the considerable, and vital, work peer reviewing involves.

Despite its many benefits, a transparent peer review process also presents disadvantages. If open identities circumvent problems with double-blind peer review, they have the very flaw of single-blind peer review the double-blind process is intended to address – namely, the possibility of implicit bias. Proponents of double-blind peer review would no doubt point out that the accountability and transparency of open identities are no protection against implicit bias, precisely because implicit bias is, by definition, not transparent and hard to prove. Furthermore, transparency opens up all reviewers to the possibility of influence or reprisal, and poses a particular challenge for early career academics, who may be reluctant to criticise work by well-established researchers. This allusion to the hierarchies within academia raises another question: how often does open peer review actually lead to exchange and debate? We should not forget that underlying the peer review process is a relationship between an assessor and an assessee, and that an early career researcher may hesitate to engage with a senior colleague. Finally, there is the danger that open reports have a ‘newspaper headline’ effect that may push readers to see flaws in a piece of work that ordinarily they would have thought unproblematic or would not have picked up at all.