Designing a new research assessment and funding system
Most countries’ research funding systems have two parts: institutional funding that pays for infrastructure and staff; and competitive funding from external sources such as research councils, agencies and foundations. This allows research organisations to pursue long-term strategies while incentivising the pursuit of quality and balancing societal needs with basic research.
Like many countries, the Czech Republic had adopted a performance-based system for allocating institutional research funding, in an effort to improve performance. Unlike others, the Czech system covered all state-funded research organisations, not just universities. Its mechanical way of working earned it the nickname ‘the coffee grinder’, and it suffered from gaming, erratic outcomes and a one-size-fits-all approach to funding – inadequate for the diverse range of universities and institutes it affected. Like most performance-based systems, it also failed to provide useful information for researchers or policy-makers about how to improve. Following criticism in the 2012 International Audit of the Research, Development and Innovation (RD&I) system in the Czech Republic, it has been put on hold.
A new approach to research assessment
A new system, the National Evaluation of Research Organisations (NERO), should start operating soon. It has been designed from scratch to address the former system’s weaknesses and to integrate learning into the assessment process – a new departure from much international practice.
The Czech Ministry of Education commissioned Technopolis Group to design the new system. Our support started by defining the key design principles, including the need for fairness and transparency, and the importance of taking the diversity of actors into account – from their scientific fields to the specific missions of different categories of research organisation.
NERO clearly separates assessment, i.e. judging how ‘good’ research performance is, from the funding process. The unit of analysis is a ‘research unit’, meaning a group of researchers working in one research-performing organisation on a specific field — this may be a group, department, faculty or other entity. The amount of institutional funding an organisation receives depends on the aggregate performance of its research units.
In addition, NERO is based on international peer-review, to obtain an independent but nuanced analysis of performance and provide feedback on how to improve it. The peer reviews are by field-specific panels to ensure they are qualified to make their judgements.
Research units write self-assessments and provide a sample of their best outputs. The peer reviewers use these, along with bibliometric indicators, to assess each research unit’s performance using five criteria. The assessment involves not only scoring but also explaining judgements and offering suggestions for doing better. The unit assessments can be combined by field to provide an overview of its health in the Czech Republic, or by organisation to show the performance of whole faculties or research organisations.
Funding for continuity In the new funding model, the scores given by the peers for each of the five criteria has an effect on the research unit’s performance-based funding, but there are separate ‘pools’ of money for universities, research institutes, technology organisations and other categories, such as research-performing museums. Overall institutional funding, conceptualised in the context of a long-term working relationship between a ministry and a research organisation, has three components:
- A performance-based component;
- A block grant, which is proportional to the block grant plus the performance-based funding that the unit received after the previous assessment;
- Funds based on a performance agreement with the ‘owner’ of the unit. This enables the owner to help the unit invest in new directions or to set up new units in organisations that do not yet have the history needed to obtain performance-based or block funding.
Qualitative assessments for improved performance
NERO avoids setting different scientific fields and research organisation types in competition with each other, recognising the missions of different research organisation types in society.
A key value of NERO lies in its formative function, i.e. in the development of strategic information. In contrast to the previous system, it is not an arithmetic exercise and the methodology stresses the importance of qualitative information to inform both the evaluation panels and the actors in the RD&I system responsible for the definition of policies and strategies.
Implementing NERO can be accommodated within existing organisations and structures. Deciding the relative weights to assign to the different assessment criteria, and the allocation of performance-based funding among organisation types, are the first challenges. The values used can be changed over time but must reflect policymakers’ objectives for the maintenance and improvement of the Czech research and innovation system.
