Technopolis completes two studies on the UK’s Research Excellence Framework (REF) 2021, which will inform the country’s next national research assessment exercise in 2028

Technopolis completed two studies on the UK’s Research Excellence Framework 2021 (REF 2021). REF 2021 was the UK’s largest national research assessment exercise to date and encompassed submissions from 157 higher education institutions (HEIs) and more than 76,000 researchers. It will determine the selective distribution of a substantial volume of institutional research funding, which could amount to more than £15bn between 2022/23 and 2029/30.

Technopolis analysis of REF 2021 outputs explored ways in which the micro data generated by the latest cycle might yield insights into the health of disciplines or research areas. The study included four areas of experimental analyses of REF 2021 data:

• Analysis of medical sub-disciplines to pilot an approach to categorise outputs submitted to the REF in a way that is meaningful, but distinct from the Units of Assessment the outputs are submitted to, and provide subsequent analysis. We found that this approach successfully provides a more granular view into the REF submissions to health related units of assessment (UoAs 1–4), with ‘Neurology and Neurosurgery’, ‘General and Internal Medicine’, ‘Oncology and Carcinogenesis’, ‘Cellular and Developmental Biology’, and ‘Experimental Psychology’ appearing amongst the top 20 medical subfields (the latter representing 76% of all UoAs 1–4 outputs)
• Analysis of ageing and gerontology research to test a methodology to identify outputs relating to ageing and gerontology research across all panels and UoAs, and provide subsequent analysis. We found that publications in this area are more interdisciplinary than the global average based on two interdisciplinary indicators (which capture disciplinary diversity of authors and disciplinary diversity of references).​ There is also high knowledge uptake beyond academic research – higher citations in patents, policy-related documents, practice guidelines and news items above the world average, and above the scores for the UK and UKRI-funded publications in the field
• Exploring research excellence within the arts and humanities, and the effectiveness of the REF system at dealing with a diversity of types of research outputs. We found there are no (statistically) significant differences in scores between traditional and non-traditional outputs, once we control by type of institution and HEIs’ characteristics (via econometric analysis). Overall non-traditional outputs are being submitted to REF less than would be expected. The submission of non-traditional outputs to REF2021 has decreased in absolute terms (from 5,509 to 4,436 outputs submitted) and in relative terms from 2.9% to 2.4% of outputs submitted
• Understanding the extent to which REF encourages or discourages interdisciplinary research (IDR), including its ability to assess this work fairly and robustly. Our results suggest that HEIs did not submit fewer IDR outputs than would be expected. REF outputs have similar degree of IDR in comparison with UK and UKRI publications measured in terms of knowledge integration. Research integrating a diverse knowledge base (measured as disciplinary diversity of references) was, to some degree, negatively correlated with the quality profile of REF outputs. The negative correlation between the disciplinary diversity of authors and the quality profile was less pronounced and less consistent when looking at each panel and units of assessment individually.​ The results above could be explained by IDR outputs being truly (albeit slightly) lower in quality, in comparison with average, or by a slight negative bias against IDR in the REF process

In bringing together the four analyses, we found that on-going developments in data science allow implementing automated processes (many of which rely on machine learning) to analyse certain concepts, at scale. However, the implementation of the different projects also showcases that (i) the successful implementation of data science methods requires not only the use of specialised skills but also a great of prior work in terms of structuring and curating the data information in a way that is meaningful to the analysis; (ii) these methods do not replace the need for thematic expertise and peer-reviewed assessment.

Through this project, the funding bodies gained insights into the feasibility of carrying out a more granular analysis of research assessment data, or analysis of cross-cutting themes in future exercises.

Technopolis also delivered a study to estimate REF 2021 costs and analyse the balance of costs against benefits played out in REF 2021. We estimate the total cost to the UK of running REF 2021 was around £471M, which amounts to around £3m per HEI on average for the 7-year cycle and around £6k for each researcher submitted. This figure constitutes a substantial increase in the costs borne by the higher education (+50%) community when compared with the costs of submitting to the preceding national assessment exercise, REF 2014. The overall cost amounts to around 3-4% of the funding expected to be distributed, which is more economical than the costs of the peer-review procedures (c. 12% of funds distributed) used by the grant-awarding research councils.

The higher cost of REF 2021 was driven in large part by an expansion in the scope of the exercise and in particular by the inclusion of a substantially larger cohort of researchers (+50%). From this perspective, the increased costs are in part mirrored by increased benefits in the shape of a more inclusive process that has provided greater equity to individual researchers. UK HEIs value many of the cultural shifts resulting from these changes, such as greater ability to deal with researchers’ individual circumstances or more open research practices. Indeed, the great majority of HEIs surveyed by Technopolis believe REF 2021 represented moderate or high value for money.

Both Technopolis studies are available here.

These two Technopolis studies are already making a difference, and have informed some of the key decisions on the high-level design of the next research assessment exercise (REF2028), the outline design for which published in July 2023 by JISC on behalf of the four UK higher education funding bodies here.

The team: Anete Vingre, Cristina Rosemberg Montes, Paul Simmonds.