It’s okay to lose confidence in your previous research

And to be open about it.

12 November 2019

Brain light/Alamy Stock Photo

You may not believe what you believed before. It might feel as if you’ve squandered your grant money on research that you no longer stand by, but that’s okay. That’s science.

That’s the message of the Loss of Confidence Project, which is encouraging psychologists to report the research findings they no longer consider to be true.

The project has yet to reap the kinds of insights that its founders were hoping for, with only a handful of psychologists submitting loss-of-confidence statements since its launch in 2018. Julia Rohrer, a psychologist at the International Max Planck Research School on the Life Course (LIFE) in Germany and lead researcher on the project, thinks this is likely due to the stigma scientists might associate with admitting their mistakes.

“It takes a lot of effort to own one's mistakes and make them public,” says Rohrer. “Hopefully, this will contribute to a broader discussion that changes how we deal with scientific errors.”

Everything in hindsight

The latest iteration of the Loss of Confidence Project is asking researchers from around the world to elaborate on why they no longer stand by their previous findings.

The team conducted an online survey in which nearly half of the 316 surveyed scientists — more than one-third of whom were psychologists — said they had lost faith in at least one of their older research findings.

The results can be accessed in a PsyArXiv preprint published on November 10.

According to the survey, some respondents initially had little confidence in the result they no longer believe in, while others said they had lost confidence in their conclusions rather than the overarching phenomenon they were studying.

What’s more, some researchers said they were no longer convinced by the specific phenomenon they were investigating, but still had confidence in the general overarching hypothesis.

Combined, the respondents had published around 8,000 papers and had collectively lost confidence in around 10% of these, the survey reported.

Among the dozen responses Rohrer and her team received to their initial call for loss-of-confidence statements last year was a statement from Raphael Silberzahn of the University of Sussex in the UK, and Eric Luis Uhlmann of the European Institute of Business Administration (INSEAD).

The duo said they had lost confidence in their 2013 study, which suggested that the meaning of a person’s name may affect which fields they end up working in, following a post-publication re-analysis conducted by another researcher.

Open about doubt

Rohrer and her colleagues asked those who reported losing confidence in their research — and those who were unsure if they had lost confidence — why this was the case. More than half said it was due to a mistake or misjudgment on their part, but some admitted to carrying out questionable research practices.

In less than 20% of cases, the loss of confidence was reported publicly, the survey found. When it was reported, it was often done so in later publications, conference presentations, or on social media, without linking back to the original papers. This makes the corrections difficult to keep track of, both by the scientific community and the wider public.

That said, Rohrer and her team predict that, in the larger scheme of things, a “a non-negligible proportion” of researchers have lost confidence in at least one of their findings at some point in their career, representing a tiny fraction of the total scholarly literature.

But it’s worth acknowledging that a researcher can lose faith in their previous research and shouldn’t have to feel guilty about it. After all, science is self-correcting, or at least it is supposed to be. Some have argued that science is broken, while others have said more specifically that scientific publishing and the process by which researchers and their work is judged is broken.

It is essential for scientists to openly report not only what they believe, but also what they no longer believe, in an accessible manner. One would hope that the right incentives are in place for academics to do so without fearing negative consequences to their careers.

It remains an open question, however, whether researchers feel it is in their interest to make their loss of confidence public, and how they should go about updating the literature.

Many will agree that retractions aren’t always the best or most appropriate way of correcting the record, but the grey area around what should be done often means nothing is done and studies are left as they are.

What researchers should do in such a scenario is to approach the journals that published their work in the first place and seek their advice. Journal editors should make a decision on a case-by-case basis on what would be the clearest, easiest, and most succinct way of communicating such disclosure statements to its readers.

Though the internet makes it easier for researchers to make their feelings public, these statements also need to be easy to track. Scholarly publishers, indexing databases, and digital repositories should take it upon themselves to shift their priorities towards fixing the scientific record.

For instance, Curate Science, a site aiming to boost transparency in science, is now implementing a functionality that allows researchers to post retroactive disclosure statements about their papers.

“Confronted with researchers who publicly admit to errors, one should keep in mind that this is not a reliable indicator that the researchers were less diligent in their work than others,” Rohrer and colleagues write in their paper.

“On the contrary, given the potential (or perceived) costs of individual self-corrections, public admission of error could be taken as a costly and thus credible signal that the issuer cares about the correctness of the scientific record.”