Don't let big names and impressive stories influence your peer review

How to avoid being star-struck.

23 January 2020

Feodora Chiosea/Getty Images

During my PhD, a journal invited me to review a paper by an impressive list of authors. The paper made broad claims about how exposure to environmental stressors at certain stages of early development affected fish behavior later in life. The paper was right up my alley and I gleefully accepted the invitation.

As I had read in numerous guideline documents for reviewing scientific papers, I evaluated the manuscript from start to finish. From my perspective, the paper added new and important information to the literature.

I recommended acceptance after moderate revision. Soon after the paper was published, some colleagues heavily criticized it, citing that the broad claims were based on low sample sizes and could not really be trusted – and they were right.

I felt embarrassed. How could I (and the other reviewers) have missed such an obvious flaw in the study?

I reflected on this for some time and came to the realization that the list of well-known, prestigious authors and the clear and convincing story that they told blinded me to the flaws in the work as an early career reviewer.

Although not everyone falls into this trap, bias towards being less critical of science from famous names or institutions does indeed occur. Knowing this from an early stage could have helped me avoid being star-struck and intimidated.

Nonetheless, I’ve learned some things from this process that I think can help early career reviewers (or any peer reviewer) avoid being influenced by an awe-inspiring story from high-profile authors.

Mix up your reading order

First and foremost, I’ve learned to not review a paper in a linear fashion.

While tips for reviewing manuscripts often suggest that reviewers evaluate a paper from beginning to end, my advice is to start by assessing the Methods section to ensure that the scientific rigor is adequate.

Evaluating a paper linearly has the potential make it seem more important and impressive than the study actually is.

By starting with the Methods section, reviewers can evaluate the appropriateness of sample size, experimental design and protocols, and statistical analyses before reading the story of how the authors interpreted data and what they think means in the big picture.

If the methods are robust and solid, I usually continue by assessing the results and the interpretation of the data, after which I will read the Introduction and Discussion.

Of course prospective reviewers will likely want to read the Abstract first to make sure that the manuscript falls within their area of expertise, which I certainly recommend. But I try to avoid reading author names before reviewing a manuscript. This helps me avoid being star-struck or intimidated by high-profile authors.

The tips above help me avoid being influenced by names and stories during peer review. I do recognize, however, that not paying attention to author identities or not reading a paper linearly can be difficult to follow in practice for some reviewers.

I think changes in editorial practices at journals can help reviewers avoid these issues as well.

Time to blind author identities?

While not a perfect solution, I think that blinding author identities can help minimize the chances of author prestige influencing the quality and rigor of a given peer review.

Pre-registration of studies can also help ensure that methodological approaches are robust before the study is even conducted.

Perhaps as a stepping stone to widespread pre-registration, I think it could be beneficial for journals and editors to consider only sending the Methods section to reviewers once they accept an invitation.

Coupled with blinding author identities, this approach would force reviewers to first focus on the quality of the science rather than the story and who wrote it.

Peer review is far from perfect and reviewers are becoming increasingly hard to find. Early career researchers make great reviewers and they can help ease the challenge of finding reviewers. Making sure that they are prepared to properly review papers from high-profile senior authors is important.

My hope is that the experiences and lessons detailed above can help prepare early career reviewers from being influenced by famous names and impressive stories.

Jeff C. Clements is a research scientist with the Department of Fisheries and Oceans Canada