Who’s Afraid of Reviewer #2?

No one is likely to argue that academia attracts many of the sorts of people who create spaces for collaboration, get involved with their colleagues’ projects, and value a supportive community that they lovingly nurture. As peer reviewers, some of these academics might even write things like ‘this journal is lucky to publish your article’, balancing praise with clearly marked-out areas for improvement. They recognise the pressures of academic life and don’t want to add to them unduly. These people – and there are plenty of them – make academia a joy.

Unfortunately, however, academia also tends to attract other sorts of people – the egos who strut around campus without a thought or care beyond their own research careers and pet obsessions. When it comes to peer review, these invariably rude, lazy, and smug academics are more likely to write things like ‘the authors seem to be reinventing the wheel and a flat tire to go along with it’. Some of these reviewers have even been found to purposefully downgrade review scores to gain personal advantage – deliberately obstructing colleagues working in the same field, or hobbling the chances of junior academics attempting to get on the career ladder. Although these academics thankfully remain a minority, there are enough of them (and they are enough of a menace) to have sparked a meme: Reviewer #2.

Reviewer #2 revels in rejecting and humiliating their peers. Academics are used to rejection, of course, and for the most part we can accept it and live with it. But rejections can also be unjust and discriminatory. Perhaps the reviewer has been unprofessional, letting bias influence them. An author’s gender, social class, nationality, ethnicity, discipline and any other characteristic might be apparent even through blind review processes, consciously or unconsciously influencing a reviewer. Reviewer #2 might also simply be unreasonable, unwilling to judge your paper on its own terms. Reviewer #2 is a meme, of course, and at least partly a fiction, but it nevertheless reflects the real dark side of the unfair standards, cultures of bitterness, and bullying that exist in academia. These problem reviewers are a real phenomenon worthy of investigation. A handful of papers, in fact, do just this.

A paper by Christopher Worsham and colleagues, entitled ‘An Empirical Assessment of Reviewer 2’, found that ‘popular beliefs surrounding Reviewer #2 may be unfounded’. But that was a small-scale study published in the British Medical Journal, investigating only papers in that journal. Iowa State’s David Peterson set out to measure the extent to which ‘Reviewer #2 crushes your hopes’ in a far more colourful paper called ‘Dear Reviewer #2: Go F’ Yourself’. He, too, found no evidence of particularly bad behaviour from the people officially designated as Reviewer #2. There are bad reviewers out there, he concludes, but they don’t necessarily occupy the number 2 slot. But (…wait for it…) in some disciplines Reviewer #3 was indeed a negative outlier. Statistically, they are ‘significantly more likely to be the reviewer, based on the overall evaluation, who dooms a manuscript’, writes Peterson. Perhaps, he supposes, the third reviewer slot might be taken by a time-poor academic, who is either too stressed to respond to the editor in a timely fashion or has received too many annoying emails from an editor, goading them into giving a hostile review. It’s difficult to tell.

Even if the academic literature on the subject hasn’t strictly managed to observe the Reviewer #2 phenomenon, Reviewer #2 (or #3) has taken on a life of its own. And there’s widespread support for the idea that Reviewer #2 must be stopped. There is even a Downfall meme, and a Facebook group, which has, as of May 2023, nearly 126k members. It grew, in fact, by more than 3,000 members just while we were writing this blog post. How, though, do we stop Reviewer #2? First, we must acknowledge an awful prospect: according to Assistant Professor of Sustainability at ESCP Business School Gorgi Krlev and Dean and Professor of Organisational Behaviour at City, University of London André Spicer, we each have the potential to become Reviewer #2.

Epistemic Respect

Krlev recently had a brush with Reviewer #2. The journal’s final decision on his paper was a rejection, but he didn’t have a problem with that, per se. It ‘happens to every researcher, all the time’, he writes. But Krlev felt that Reviewer #2 was being unreasonable and unprofessional. ‘You’re citing papers in lower tier journals’, the reviewer wrote. ‘It doesn’t surprise me’ they went on, ‘that authors in lower tier journals are making this argument, but that doesn’t make it correct’. Krlev’s tweet about this review went viral. It led to him promoting the concept of epistemic respect along with Andre Spicer – an approach to peer review that we should all embrace.

Epistemic respect involves ‘making a conscious effort to understand and appreciate [a] claim, even though one might disagree or have difficulties relating to it’. You should, Krlev and Spicer argue, try to understand a claim without wilfully or carelessly misrepresenting or caricaturing it. You should judge a claim on its logical soundness, the clarity of the argument, and the empirical support an author brought to their work. It is not OK, by this measure, to weigh a claim according to the prestige of an academic’s institution, or the journals an author has published in. And, more generally, a respectful reviewer should engage with an article on its own merits, acknowledging strengths and limitations in a balanced way.

Epistemic respect starts, perhaps, by asking yourself if you’re the right person to review a particular paper at all. If you don’t have the appropriate disciplinary background, or the requisite knowledge about a particular methodology, don’t accept the job. Similarly, don’t agree to review a paper if you don’t have the capacity to do a good job. Just as you shouldn’t go to sleep after an argument, or run a 10k race after a nut roast, you shouldn’t try to review someone else’s work if you can’t give it the necessary time it deserves. If you don’t recognise any of the names in the bibliography (or simply don’t have the time), you have two choices: politely decline to review the paper, or accept the job anyway, then insist that the writer doesn’t know their onions. Three guesses which of these responses is acceptable.

When we accept a manuscript for review, our advice on giving and receiving feedback will also help keep our own inner Reviewer #2s in check. We look at peer review as an opportunity to maximise positive comments, by focusing on theme, the completeness of a literature review, or the suitability of methodology, for instance, and all of our comments should be specific. Where something is missing, we should indicate what this is and why it is important to include it, eliminating any comments that were written essentially to make us feel superior.

As referees, we have a responsibility not to become Reviewer #2. As authors, we have a right to expect journal editors to protect us from them.

Stopping Reviewer #2

One way of stopping Reviewer #2 is to experiment with various levels of open peer review, directly engineering closer contact between reviewers and authors. It isn’t surprising that closed and blind reviewing promotes bad behaviour. As recommended by the University of Greenwich, feedback should ideally be a dialogue. Openness and visibility promote mutual respect. You’re less likely to say something like ‘nothing is right with this article’ if its author is sipping an espresso in front of you. The best way to stop Reviewer #2, then, might be to move towards more open forms of review. In 2016, Nature introduced transparent peer review, which allows authors to publish comments received from reviewers, alongside their own responses. The transparency only goes so far, because ‘referees will still have the option to remain completely anonymous’, but this is still a step towards a more open system, built on mutual respect.

A more obvious solution would be to step in when a reviewer submits unwarranted, rude and discriminatory criticism. As one commentator in Krlev and Spicer’s study put it, ‘a handling editor should be more than just a post office’. Abusive reviewers have been known to somehow work out who wrote a given paper, then trash it. Editors need to become better at spotting abuse and resisting Reviewer #2 when they crop up. It’s not always easy to confront someone who has voluntarily given up their time for no pay, but if their comments are obviously unjust or abusive, editors and associate editors have a duty to prevent them from reaching the author. Indeed, the journal’s final verdict should never give Reviewer #2’s report a weighting it doesn’t deserve.

Tesearchers who are looking for incisive feedback on their work before sending it off for peer review may also wish to commission a Journal Suitability Reporting service. This seeks to pre-empt the kind of assessment you may receive from any particular journal and minimises run-ins with Reviewer #2.