Scientists go to great lengths in reviewing high-quality research

An analysis of review length raises questions about what quantity means for quality.

11 April 2018

akindo/Getty

Peer reviewers write 300 more words for high-quality research journals on average than they do for other journals, according to an analysis of 378,561 scientific peer reviews tracked by the online database Publons.

Compared to the average review length of 342 words in 2017, those for the subset of journals in the Nature Index were 640 words.

Andrew Preston, managing director of Publons, says that reviewers may be putting more effort and time into writing extensive reviews for higher-quality journals to please editors.

Giving a review that extra bit of attention “can help forge much-needed connections with editors,” Preston says. “This could increase a reviewer’s own chances of publishing in that journal.”

The European Physical Journal C, The Journal of High Energy Physics and Astronomy & Astrophysics had the longest reviews on average, while reviews for Chemical Communications, Journal of Physical Chemistry Letters and Chemical Science were below 400 words.

While review lengths have decreased over the past year, the decline is less marked among the index group of journals.

A separate Publons analysis of reviews in 2016 and 2017 found that word-length increases as a journal’s impact factor increases. Reviews for journals with a high impact factor largely exceeded 400 words.

The team at Publons is asking reviewers, authors and editors to rate the quality of reviews to gain further insights into the relationship between review quality and length.

While review length alone cannot be used to determine quality, Macquarie University developmental psychologist, Nicholas Badcock, says that the analysis could be helpful for less experienced reviewers.

“Many early-career researchers fret over their reviews and end up writing comments that compete with the article word count,” says Badcock, who has written more than 80 reviews. “This analysis presents a benchmark length to aim for.”

When the Publons team looked at reviews from different countries, they found that scientists in Western countries used more words than those from Asia and the Middle East.

Researchers in Iran, China and Egypt wrote reviews that were half the length of those by researchers in Greece, the Netherlands and France, who produced the longest reviews.

Review lengths also varied across different disciplines, and were longer in journals that predominantly published on discussion-based subjects, such as psychology, compared with more numerical fields, such as mathematics.