The trolls have infested academic peer review

18 Comments

I long ago gave up blind submissions to journals where I am not known. Fortunately for me, there are many other ways to publish strong work (outside of the sciences and social sciences, at least). Open review is the answer, I think: both the author and the reviewers should know each other’s identity. That is the only way to ensure real accountability. Many European publishers used to employ a similar process, though “double-blind” review has been forced on them by the administrative needs of faculty members in the Anglosphere. And BTW, I started submitting articles well before the internet age and reviewers’ comments were often just as irrational, bloody-minded, uninformed, misguided, belligerent, disdainful, disrespectful and downright ornery then as now!

I agree with Andrew Gow’s comments, and have had similar experiences. One SSHRC project was (narrowly) rejected I believe on the basis of one extremely negative review that included sentence fragments, many exclamation points, and very little reference to what the project was about.. Since the other review was very positive, i can only assume that the committee, needing a reason to eliminate some proposals, responded to the negative review without fully endorsing it.

I have also had excellent experience with reviewing and being reviewed by journals with an open review process.

Blind reviews are often just fine –written by reasonable and responsible people who contribute to scholarship. But I have also seen some humdingers. I doubt sane people would put their names on them. The fetish for blind reviews needs to be held up to some serious scrutiny. It is a technique, and only a technique, and I don’t think it works.

I have been publishing for far longer, in the life and earth sciences; first paper in 1987. As an author I too have seen much boorish, arrogant, and hypocritical reviewing, and as a reviewer have seem equally arrogant, boorish and hypocritical responses by authors to my reviews. I have also seen both very ethical and credible reviews and reviewing as an associate editor for a journal, and some truly good handling of poor review practices by editors. However, I have also seen editors ‘drop the ball’ and let bad review practices happen, and have resorted to calling them on these lapses – not always with success.

As a consequence, I do think the blind review (‘anonymous review’ as its more commonly called in the physical sciences) has had its day.

Some European journals, such as those published by Copernicus Press (primarily in English), follow an open review model where anyone in the discipline can post comments on an article under review (published first online as a Discussions paper), and the 2-3 official reviewers also have their reviews published online, along with the authors’ responses. From what I have seen of these – as both an author and a reviewer – is that the informal and formal reviews are collegial, constructive and helpful to the authors. And as far as I can tell, there is no diminishment in quality, and papers in these journals command high citation rates and other impact metrics. For my graduate students, they say this is a far more attractive opportunity for publishing, and a far more positive experience.

Ryan. I agree with you. I have written along these lines in University Affairs: “Time to rethink peer review
Evaluating scholarly work in the Internet age.” By MURRAY DINEEN | DEC 05 2012. While my interests are those of the humanities and the arts, I believe it is particularly important for the sciences and for legal and political studies, where research must be evaluated fairly and in a timely fashion. I will conclude simply by pasting an excerpt of my thoughts as published in 2012, and draw your attention to the work of Rosanna Tamburri, from 2012, cited in my article.

“The Internet allows for timely and humane forms of exchange in scholarship. In the hands of an editor, peer review could become a form of colloquy, an exchange between author and reviewers. “Open peer review” and “open peer commentary” should become fully accepted practices of scholarly review.”

In four decades of submission of scholarly work, I have never received the sort of snide review that Ryan Bullock mentions. Nor have I ever written this sort of review. Anyone receiving such a review should complain about its content to the editor who selected this reviewer, attaching a copy of the review. Any editor worth their salt would immediately stop using such a reviewer, and would disount their recommendation.

That is theory. In practice it does not work that way.
The reports can be so devious and underhanded that you even do not find the words to answer.
If you just tell the truth it turns the tables on you. You have never been bullied.

I think anyone that has ever submitted a manuscript for publication has had some bad experiences with reviewers. My concern with a completely open review process would be that reviewers may concerned about repercussions of a critical review. You can be critical and maintain a tone of civility at the same time. Reviewers may also be inclined to give a more generous review with the expectation of reciprocity in review of their own manuscript or grant proposal.

Ultimately, it is the editor should be able to evaluate reviews and throw bad ones out or arrange for additional reviewer. Unfortunately, I think editors often fail at this point as well.

Not sure that there is a good solution.

Thanks Ryan, that’s a great piece!
A few years ago I participated to a review of peer review systems where we brushed potential future systems based on existing review systems outside of academia. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5686505/

I’m sure many will have particularly funny examples of outrageous comments from reviewers. My own personal favorite was a reviewer who began with a number of vague and negatively toned remarks which were so off the mark that it seemed as if I had received a review of someone else’s submission in error. Then he or she went on to take issue with my spelling of “judgement”. Rather than recognizing that “judgement” and “judgment” are acceptable variants, with one perhaps being more common in the US or Canada than the other, the referee claimed that there was a difference in meaning and my variant made no sense in the context. This was followed with some insinuations about my educational background.

I am an Emeritus Professor. A major part of the problem lies with the Editors and the Editorial Board. Most journal editors leave the review process to be handled by their secretaries (or executive assistants). The secretaries routinely just “forward” the referee comments to the authors. They are tasked with rejecting a paper if the comments are not fully favourable. If journals honestly care about the review process, the editors should boycott any referee who makes such comments as those the author of this article mentions. Further, the editors should also penalize such unethical referees by barring them from submitting their own papers to the journal for a period of two or three years.

My idea is slightly different. How about a system in which authors are permitted to submit their work to, say, two journals simultaneously? This would introduce some healthy competition and hopefully make editors more alert to identifying work that they may find particularly suitable for their journals. In this way, the publishing process would be a bit like a job search – it is, after all, perfectly fine and legal to apply for multiple jobs at the same time.

I would submit that the blind process itself is not to blame here. The unwillingness of editors and publishers to require substantive, professional review and commentary (lest the reviewer’s criticisms be disregarded) plays at least as great a role in this trend toward incivility. Where incoherent and scattered points of disagreement are submitted, an editor or publisher needs to ensure that the response was not submitted by some undergraduate research flunky by insisting on relevancy in critique.

I have submitted to a lot of top international journals in my field. The good ones give useful (critical) reviews, although you’ll almost always get one useless reviewer who clearly couldn’t be bothered to read the paper. However, I recently had an editor and two reviewers tell me to “get a native English speaker” to read the paper for me. As a professional copyeditor and native English speaker, I was astounded. The reviews themselves were written in such poor English, I could hardly make sense of them. I still wonder if that was a ploy to get me to pay 400$ for the journal’s editing services.

We can’t forget that escalations of poor reviewer and editorial practices are another consequences of a bloated and fragmented publish-or-perish system. No one does their best work when they are too busy, and I’ve read much from both editors and reviewers drowning in tsunamis of manuscripts. While lousy reviewers have always been a problem (previous comments attest to this), you’d have to think the quality of reviewers is going to go down as editors are desperate to get *any* reviewers. I myself have been asked to review papers I didn’t feel I was qualified to review. Anyone who is bloody fed up and cynical about the publication process (aren’t we all?) might find solace in Alvesson et al.’s (2017) Return to Meaning: A Social Sciences with Something to Say.

Another angle: I shudder to think how some of these reviewers treat their students and student writing.

An excellent piece of reflection. Very relevant!

Two bad interactions in 15 years hardly sounds like an infestation.

A troll at work
The misuse of anonymous peer review is the rule rather than the exception.
Trolling is like bullying, psychopathic behaviour.

I have experienced and witnessed the same, and agree with the author on every point presented. While the reviewers can be unreasonable, blunt and harsh, providing very little to no constructive feedback, such behaviour is enabled and even endorsed by journal editor(s). Perhaps, the discussion should also bring editors and their ethical compass into limelight.
Some, but not all, journals provide review guidelines, but rarely do they send a document, or at the very least a link outlining the best review practices along with their review request. They place onus on the reviewer to do a proper (and ethical) job. And such assumption sometimes fails everyone along the line.
Many reviewers seem to forget some of the key points of a review – focus on merit and scientific value, help the authors improve their work, and most importantly, provide constructive feedback. The latter is of utmost importance. I am sure that most of those who went through the effort of preparing a manuscript would want to know how to approach the deficiencies and improve the work, so the next iteration can be even more worthy of publication. Once published, such paper will garner more attention as well as citations. I have seen both ends of the spectrum; editors who genuinely care and always try to ensure they obtain just and constructive reviews, and those who evidently have a difficult time navigating through the scope of responsibilities their position carries. Journal editors have the power to ensure that the review process is fair and void of trolling, and we should not forget to bring them into the discussion as well.