What's the best measure of research output?

21 March 2016

Danqing Wang

An artist's interpretation of a paper published in Science that compared human and machine learning.

EXPLAINER

When ranking countries based on their output of high-quality research, weighted fractional count (WFC) is often used as the Nature Index's primary metric. And for good reason. WFC reflects the size of the contribution a country's researchers have made to every study published in the 68 top-tier journals included in the index. This measure also takes into account the higher proportion of astronomy papers in the index. It seems astronomers love to write papers. The index's unweighted measure of contribution is fractional count (FC).

For every paper included in the index, the FC and WFC are split among authors based on their affiliation. Take a recent paper published in Science, which created a computer model that captures humans' unique ability to learn, and had three authors from three different universities, two in the USA and one in Canada. For this paper, each affiliation received an FC of 0.33, and because it wasn't published in an astronomy journal, they received the same WFC. Adding the WFC of a country's institutions presents a picture of that nation's performance over a designated period of time.

In a recent post, we published a graph of the top and bottom 10 countries ranked by change in their WFC. It revealed a compelling narrative. Since 2012 China's contribution to high-quality research has soared, while traditional stronghold, the United States, appears to have lost its mojo. (Although it's worth noting that the USA's total WFC is miles ahead of anyone else, China included.)

But what does it mean when a country's WFC drops? Does that suggest its research performance is slipping?

Not necessarily. When assessing the output of a country's top-quality research, it is prudent to also consider article count - the total number of studies that a country's researchers have contributed too, regardless of the size of that contribution.

Consider this next graph. It shows the change in article count for the countries in the graph above. While the United States and Japan's article count followed a similar downward trajectory to their WFC, the article count of all the other countries grew between 2012 and 2015 - including the eight countries that experienced a drop in their WFC.

As article count isn't a weighted metric, it shouldn't be directly compared to WFC. When considering the change in a country's total number of papers versus the contribution it made to those papers, it best to compare AC with FC.

An interesting trend emerges when a country's article count goes up, but its fractional count dwindles. It suggests that while the country's researchers have contributed to a larger total number of studies, the proportion of their contribution has become smaller.

The reverse can be observed in countries with an increase in their FC but a fall in their AC. In those countries, researchers contributed to fewer studies but received more of the credit for the ones that were published.

These two scenarios at least partly reflect patterns of collaboration, and their significance will depend on a broader research context. What is certain, however, is that neither WFC, FC or AC alone can reveal the state of a country's high-quality natural science output. The three metrics should be considered together.