Why digital badges don't work

Turns out they're not enough incentive for researchers to share their data.

18 March 2020

Open Science Collaboration/CC BY 4.0

Online badges don’t motivate researchers to share their data when submitting to a medical journal, the first ever randomized controlled trial on the practice has found.

The study, published in Royal Society Open Science on 18 March, looked at whether researchers submitting manuscripts to BMJ Open are more likely to share their data when presented with the opportunity to earn a digital badge on their paper.

The researchers behind the study, led by Anisa Rowhani-Farid, a meta-science researcher now based at the University of Maryland in Baltimore, contacted the authors of 160 papers undergoing peer review at BMJ Open; half were offered a badge for sharing their data, the other half were asked to share data without being offered an incentive.

Of the 80 studies whose authors were offered a digital badge, 57 were published after going through peer review, but authors of only two studies opted to share their raw data.

Similarly, out of the 54 papers that were accepted for publication from the other group, only the authors of two studies opted to share their data. That puts data-sharing levels at just under 4% for both groups.

Different challenges across fields

The results came somewhat as a surprise, Rowhani-Farid says, because previous studies have suggested that there may be greater effects between digital badges and data-sharing practices.

For instance, a 2016 study by researchers at the Center for Open Science (COS) in Charlottesville, Virginia, found the rate of data-sharing increased from 1.5% to 39% after Psychological Science started offering badges.

A 2018 study by Rowhani-Farid and colleagues also found there to be a 7.6% increase in data-sharing at the journal Biostatistics following the introduction of badges, but detected no change in the rate of code-sharing.

It might be that data are more difficult to share in the health and medical sciences due to patient confidentiality laws, rather than in fields like psychology, where data-sharing is more common, Rowhani-Farid says.

Norms around data-sharing also vary across different disciplines and ages, with some fields, and younger academics, more open to making data available.

Rowhani-Farid isn’t suggesting that journals should abandon badges altogether. “Let’s carry on,” she says, “but let’s be a little bit more conscious of evidence-based policy.”

Considering the public perception

Brian Nosek, executive director of COS and a psychologist at the University of Virginia, says that the study does not consider the extent to which the badges were visible to the wider research community, which he thinks is important to lead to change in data-sharing norms.

“Badges are thought to change social norms by making visible when other people are doing open science behaviors, but those sorts of interventions won’t work if you can’t see what your peers are doing,” Nosek says.

Nosek says that Psychological Science’s data-sharing rate has risen sharply since it started awarding badges in 2014, with more than 60% of studies published in the journal now including an open data badge.

“If that is robust, it could mean that norms are sufficient in this context to shift most, but not all, behavior, or that some types of data and materials in psychology cannot be shared openly to earn a badge – or both.”

Holly Murray, head of data and software publishing at the open-access publication platform F1000 Research in London, UK, says badges alone can’t solve the problem of the lack of data-sharing.

“Clearly articulating why and what we want to see from open data as a global research community can lead to the development of sensible incentives,” she says.