Putting into practice a declaration of principles to change research assessment: experiences of a university and a funding agency.
The 5th Strategies CoP meeting aimed at presenting the Declaration on Research Assessment (DORA), which addresses unintended cognitive and system biases that perpetuate inequitable review, promotion and hiring practices in higher education, research and innovation institutions.
We heard from two institutions that have started transitioning over the last few years towards an open research environment – a transition that entails reforming models of evaluation. The Health Research Board (HRB), an Irish state funding agency, signed DORA in 2017 in alignment with other measures, including an Open Research publication platform, to center importance on the intrinsic value of research and its societal impact. The Open University of Catalonia (UOC) was the first university in Spain to sign DORA in 2018, as part of wider efforts towards open knowledge and fair research principles and practices.
Dr Marta Aymerich, Vice President for Strategic Planning and Research at UOC explained how they initiated change in line with DORA principles for postdoctoral calls. Removing number of publications and journal impact factor as criteria, they are now assessing four main achievements of an applicant as a way of evaluating scientific output. Moreover, instead of looking at the research group scientific output, they now consider the mentor achievements regarding trained PhDs (for more details, consult the PPT presentation). UOC is working closely with the European University Association (EUA) to foster change and improve renewed processes. However, their room for maneuver depends largely on external change regarding dominant procedures.
Health Careers Programme Manager, Dr Annalisa Montesanti described the reforms undertaken at HRB, such as the introduction of guidance and training provided during the review process. With written guidelines and videos on unconscious bias, they seek to limit bias, including gender ones (check the raising-awareness video used on unconscious bias from the Royal Society and the Cerca Institute). Dr Montesanti also presented their narrative-based CV that does not contain any type of publication metrics (find the details of the CV on the PPT presentation). Those changes are well accepted by both researchers and panel members, she pointed out.
The discussion that followed with the participants brought up first the issue of time and workload regarding those new measures. As this is still a change in process, combatting resistances and myths surrounding the presupposed fairness of metrics as well as learning new practices may represent time-consuming efforts, the speakers concede. This is why the two institutions and the EUA are working on developing improved guidance for more efficient and impactful processes, both for evaluators and applicants.
Regarding the impact on gender equality, there is no data collected by the UOC or the HRB that could indicate positive effects. There is a need for more studies and comparisons to analyze how the implementation of DORA have impacted the outcomes of evaluation processes. However, the few evidences that exist, such as the experiment at the Canadian Institutes of Health Research, show that fairer assessment processes contribute to greater gender equality. At the HRB, Annalisa observed so far that by inserting a section on “Career breaks” within the CV, it allows reviewers to know the reasons behind productivity gaps, and may thus impact positively women who are more likely than men to have career breaks due to family care duties. Moreover, mentioning other involvements in the community of research (such as taking part of gender equality projects) accounts for a more holistic view of research and in turn can valorize those engagements on the long run.
For more information on DORA, please check the uploaded resources.