Research Evaluation SIG drop-in session on “Why all the interest in the university rankings, and what can research managers do?”

On 1 May, the ARMA Research Evaluation SIG hosted a drop-in on “Why all the interest in the university rankings, and what can research managers do?” Dr Lizzie Gadd, one of the SIG co-champions, kicked off the session with a presentation on the use (and misuse!) of university rankings, in which she outlined a number of issues starting with methodological concerns.

Firstly, what are the rankings actually trying to measure? Which university is best in the world? Best at what? The Academic Ranking of World Universities uses the number of alumni winning Nobel Prizes and Fields Medals as the sole indicator of the quality of education, which most would agree is an extremely limited indicator! Other issues such as data quality and sensitivity have been highlighted by the sector. Beyond methodological issues, international university rankings can embed inequalities. Due to the methodologies used, highly ranked universities are typically old, large, wealthy, research-intensive, science-focussed, English-speaking and located in the Global North. Moreover, the rankings maintain status quo, concentrate power and prestige amongst universities at the top, and promote competition over collaboration by suggesting that there can only be one winner.

So what can universities do to respond to these issues? Some universities, including Yale, Harvard and Utrecht University, have decided to boycott rankings citing questionable methodologies and conflicts with their values. Recognising that boycotting rankings isn’t feasible for a lot of organisations, some recent challenges to rankings from the EUA, the Dutch Universities and a UN thinktank have encouraged organisations to challenge the rankings by educating stakeholders about their limitations, making submitted data publicly available to reduce commercial power and supporting better ways of assessing university performance (e.g. U-multirank (quantitative) and More Than Our Rank (qualitative)).

At this point the discussion was opened up to the SIG community, using a mentimeter run by co-champion, Grace Murkett, to support participation. 92% of those who responded said their university had a KPI related to international or national rankings. Of those that didn’t, it was noted that there can be an ‘unspoken’ use of rankings in institutional strategy nonetheless. 43% of respondents said their university “loves” rankings, though the staff within institutions often didn’t and sometimes university leaders themselves felt conflicted. When asked how their university uses rankings, the most common responses were ‘marketing’, ‘benchmarking’ and ‘recruitment’, which highlights that often the use of rankings is beyond the remit of research managers and administrators, falling more into the workload of planners, communications teams and HR. Some areas, such as the use of REF results, were seen as easier for research managers and administrators to influence.

It was agreed that those who don’t have the privilege to boycott can still contribute to the collective power of universities to challenge the quality and robustness of rankings, given their pervasive use across the sector. Attendees gave examples of actions they had already taken to encourage a more critical response to rankings at their university, including developing a responsible research assessment policy and/or associated resources, adding caveats and content to rankings when used, and integrating these considerations into research culture initiatives. When asked what they might do in future, attendees suggested signing More Than Our Rank and CoARA, challenging colleagues on whether the use of rankings aligns with equality, diversity and inclusion initiatives, influencing the next generation of academic leaders through grassroots activities, and encouraging senior leaders to express concerns directly to ranking organisations.

The Research Evaluation SIG co-champions would like to thank everyone who participated in this interesting and fruitful discussion. We hope to see you all at the ARMA conference. In the meantime, we encourage anyone who hasn’t already joined our SIG mailing list to sign up to hear about future drop-in sessions!