Steps to a top-scoring impact case study

Describing the benefits of excellent research in simple language gets high marks in the REF.

12 July 2018

COMMENT

Mark Reed et al.*

z_wei/Getty

There is an art to writing a strong research impact case study.

The four points below, gleaned from our analysis of the United Kingdom’s Research Excellence Framework 2014, may help researchers excel in the next assessment, due for 2021.

But first, some background. In 2014, the UK became the first country to comprehensively assess the impact of its research as part of a national assessment. Although individual case scores were not made public, for our analysis we looked for institutions whose case studies were all given grades in the same range to identify 175 (out of 7,000) high- or low-scoring case studies from a cross-section of disciplines.

We combined qualitative thematic analysis with quantitative linguistic analysis to explore what made a high-scoring submission.

In a nutshell, high-scoring case studies went beyond merely describing their research; they provided evidence of the significance and reach of the benefits it led to. The case studies were also written in a more direct style.

However, new rules are being introduced for the next REF and best practice is changing rapidly in some areas. This could mean sticking to best practice from REF2014 may not be enough to reach top scores in the next REF. Depending on the approach that REF2021 sub-panels take to scoring (whether the number of cases getting top scores is unconstrained or efforts are made to avoid awarding significantly more top scores than REF2014), then competition for top scores may be fierce. Following best practice from REF2014 may not be enough to reach top scores in the next REF.

1. Articulate how specific groups have benefited and provide evidence of significance and reach

• 84% of high-scoring cases articulated benefits (compared to 32% of low-scoring cases). Those that did not articulate benefits typically focussed instead on pathways to impact.

• High-scoring impact case studies described on average 2.8 impacts, compared to an average of 1.8 impacts described by low-scoring case studies. However, in the qualitative analysis, there were a similar number of high-scoring case studies that were considered to have reached their high score due to a clear focus on one single, highly impressive impact, compared to those that were singled out for their impressive range of different impacts.

2. Give no grounds for doubt that the research was at least of internationally significant quality (warranting two stars or higher)

• 98% of high-scoring cases gave no grounds to doubt the quality of the underpinning research (compared to 80% of low-scoring cases).

• In the low-scoring cases, grounds for doubt were found in the resulting publication, funding sources, narrative description and contribution of the researcher to underpinning research.

3. Establish links between research (cause) and impact (effect) convincingly

• 97% of high-scoring case studies clearly linked the underpinning research to claimed impacts (compared to only 50% of low-scoring cases).

• Quantitative linguistic analysis showed high-scoring case studies were significantly more likely to include the phrase ‘led to’ linked to specific impacts, compared to low-scoring case studies, which were more likely to use the phrase ‘a range of’ and the word ‘improvement’, often linked to non-specific, indicative lists of benefits or beneficiaries.

4. Make sure the case study is easy to understand and well written

• 73% of high-scoring impact case studies were considered easy to read for a non-specialist audience (compared to 53% of low-scoring cases).

• Quantitative linguistic analysis showed high-scoring case studies were significantly different from the underpinning research they cited in their choice of words. They used more direct, plain language, compared to low-scoring case studies, which have many stylistic features in common with their underpinning research, such as expressing uncertainty by hedging statements.

These findings are based on a preliminary analysis of data presented at the INORMS 2018 conference for research managers and administrators, held in Edinburgh on 4–7 June.

The full version of this article is available on the Fast Track Impact blog.

*About the authors

Mark Reed is a professor of socio-technical innovation in the School of Natural and Environmental Sciences at Newcastle University.

Bella Reichard is a PhD researcher at Newcastle University, where she investigates the language of impact case studies, comparing high- and low-scoring case studies.

Jenn Chubb is a postdoctoral research associate at the University of Sheffield working on research policy through the development of a pilot study exploring academic attitudes towards the REF.

Ged Hall provides training, development and coaching in impact for academics at the University of Leeds.

Alisha Peart is research impact manager at Northumbria University, where she is developing and delivering strategies to embed impact in the research culture, and improve research impact outcomes.

Lucy Jowett is research impact manager at Northumbria University, where she works to improve research impact outcomes.