The problem with chasing publications
National initiatives that track people, rather than papers, will lead to better science in the United States.
16 November 2017
Jurgen Ziewe/Ikon Images
Science in the United States is in a much healthier condition than publication metrics indicate. A group of US universities is developing new ways of assessing science that will better depict strengths and weaknesses, and lay the foundation for a healthy research ecosystem.
Chasing the paper trail of publications has led to the overproduction of articles with spurious analyses, the proliferation of meaningless university league tables, and policy recommendations based on flawed calculations of funding inputs and publication outputs, such as the US National Institutes of Health proposal to cap the number of grants an investigator can receive. A new framework will instead trace the scientific and economic contribution of researchers.
Baseball enthusiasts will be familiar with a similar application of sabermetrics. In the late 1990s, Billy Beane, general manager of the Oakland Athletics, employed statistical methods to determine how to build winning teams based on finding the right players. These insights yielded high returns on his investments. The Athletics rose up the league, spending considerably less than teams with comparable records. What Beane did for baseball, John Marburger proposed for science. In 2005 Marburger, then science advisor to President George W. Bush, challenged the scientific community to develop better benchmarks for assessing R&D spending. Science agencies had long struggled to convince Congress, and the public, of the impact of investment using conventional metrics, such as published papers, citations, patents, or awards. Marburger argued that scientists needed a more rigorous approach to quantifying the social and economic returns of funding.
Marburger’s urging spurred collaborations between federal agencies and research universities to develop theories and analytical tools for assessing science policies. They concentrate on people, who conceive, instigate and execute new ideas.
The Institute for Research on Innovation & Science (IRIS) is building a comprehensive data platform to track the flow of these highly skilled individuals into the economy. UMETRICS (Universities: Measuring the ImpacTs of Research on Innovation, Competitiveness and Science), combines data on the labour, goods and services purchased by universities with US Census Bureau data on the earnings and employment of researchers, as well as the activity of the firms they work for. Currently, 62 campuses have committed to participating in IRIS, but membership is expected to grow to 150 universities by 2020, capturing 90% of federal spending on university R&D.
IRIS has an annual budget of about US$2 million, which will double to US$4 million by 2020. That’s cheap when compared with the US$53 million spent annually (for six years) by the much smaller higher-education community in the United Kingdom on its 2014 impact evaluation exercise, the Research Excellence Framework.
The UMETRICS platform is not just used for evaluation. I am among about 70 researchers affiliated with IRIS analysing the data to get a better understanding of the role of science in society. Results are beginning to reveal how scientists support the economy. Research-trained PhD holders moving into industry are more likely than other workers to join younger, larger, R&D intensive, high-wage firms in high-skills industries — factors associated with productivity. Adding one research-experienced worker to a start-up increases its five-year survival rate by nearly 42% and more than doubles the likelihood of its going on to employ at least 10 people.
Researchers at the University of Maryland and the University of North Carolina at Chapel Hill are calculating the proportion of UMETRICS-tracked researchers flowing to specific industries, which could identify sectors where highly skilled workers are needed. Sociologists and economists at the National Bureau of Economic Research are using UMETRICS data to analyse the impact of education and training on scientists’ earnings at all levels, which should help quantify the demand for PhDs in the workforce.
In June 2017, the Census Bureau’s Federal Statistical Research Data Centers released the matched Census–UMETRICS data to the wider research community, which should reveal many more insights. Recent work showed, for example, that while early-career female researchers earn 31% less than male researchers overall, the gap disappears when controlling for the impact that marital status and children have on women’s pay.
Investment in science can take years to bear fruit, but I am optimistic that sensible measurements of US science and scientists will help both flourish.
Julia Lane is a professor at the Wagner School of Public Policy and Center for Urban Science and Progress at New York University.