Chatter makes popular metric unreliable: study

An analysis of a popular reputation metric concludes it relies too heavily on social interaction.

29 September 2017

Mopic / Alamy Stock Photo

Researchers can easily game the metric developed by ResearchGate to boost their research profile, a study has found.

The authors of the paper, published in Scientometrics, have cautioned institutions against using the site’s RG score as a measure of academic performance.

More than 13 million users have signed up to the online platform since its launch almost ten years ago, making it one of the most popular scholarly social networks.

The RG score, the website’s own metric for scientific reputation, is based on a composite score that considers citations and publications, as well as website engagement metrics such as profile views and participation on ResearchGate’s discussion platform. The site does not disclose the algorithm it uses to calculate the score.

Enrique Orduna-Malea from the Polytechnic University of Valencia, Spain and his colleagues noticed, however, that the RG scores on some researchers’ profiles did not reflect their academic history. Investigating the score further, Orduna-Malea and his colleagues reviewed 104 researcher profiles with RG scores over 100, or particularly high scores in, for example, their number of publications or responses to comments posted on ResearchGate. The authors compared the number of followers, publications, citations, and comments and responses they gave on the website with the ResearchGate profiles and RG scores of 73 Nobel Prize winners.

The authors found that the RG scores of many of the 104 researchers had little association with their number of publications, citations or their h-index — a quantitative measure of a researcher’s productivity and citation impact. Instead, many of the high RG scores were based on the researchers’ participation on the site’s discussion platform. In contrast, the main contribution to the Nobel Prize winners’ RG score was from their publications. Their scores rarely exceeded 50 points.

One academic profile with an RG score of 439.82 had 92% of the score derived from comments the academic had made on the discussion board. This is in contrast to George Smoot, recipient of the 2006 Nobel Prize in Physics, who had an RG score of just 46.63 with 100% of the score derived from his 598 publications. Active engagement with other users on the website can inflate the RG score, Orduna-Malea says.

He believes ResearchGate is a valuable tool, especially for finding collaborators, for which he has used it himself. But, he says, when a researcher’s RG score is primarily built on social interactions rather than publications or citations, it should not be considered a valid measure of academic performance. “If RG scores are being used for research evaluation, this is very dangerous.”

While Orduna-Malea is not aware of any such use of the score, he says many academics regard ResearchGate profiles as an extension of their curriculum vitae and use it to boost their professional profile.

This is not the first time ResearchGate’s lack of transparency over the RG score has come under scrutiny. A smaller 2015 study reverse-engineered the RG score and found significant weight was given to Journal Impact Factors, which have been heavily criticised by the research community for their use in evaluating individual scientists.

Elisabeth Lex from the Graz University of Technology, an author of the 2015 study, says ResearchGate’s failure to reveal its algorithm is contrary to the Leiden Manifesto, which states that evaluators of metric-based assessments should “keep data collection and analytical processes open, transparent and simple”.

The Scientometrics paper has several limitations. The samples were not randomised. It is also no longer possible to replicate studies that used ResearchGate data, including Orduna-Malea’s, as the social media platform has removed from profiles some of the measures that contribute to the RG score, including impact points and profile views of other profiles. It has also removed the advanced search feature used to search for author-level metrics.

ResearchGate did not respond to requests for comment.