COARA will not save science from the tyranny of administrative evaluation

TL;DR

This paper critically examines CoARA's research evaluation reforms, arguing that its reliance on expert-driven metrics and limited public input risks perpetuating the tyranny of expertise and ignorance, thus failing to truly reform science evaluation.

Contribution

The paper offers a philosophical critique of CoARA, highlighting its potential to reinforce existing power structures and the challenges of aligning evaluation practices with democratic principles.

Findings

CoARA's approach is susceptible to expert bias and lack of public deliberation.

Current evaluation criteria are set by institutions, not by broad societal consensus.

Limited scope of administrative evaluation may undermine scientific diversity.

Abstract

The Coalition for Advancing Research Assessment (CoARA) agreement is a cornerstone in the ongoing efforts to reform research evaluation. CoARA advocates for administrative evaluations of research that rely on peer review, supported by responsible metrics, as beneficial for both science and society. Its principles can be critically examined through the lens of Philip Kitcher's concept of well-ordered science in a democratic society. From Kitcher's perspective, CoARA's approach faces two significant challenges: definitions of quality and impact are determined by governments or evaluation institutions rather than emerging from broad public deliberation, and a select group of scientists is empowered to assess research based on these predefined criteria. This creates susceptibility to both the ''tyranny of expertise'' and the ''tyranny of ignorance'' that Kitcher cautions against. Achieving Kitcher's ideal would require limiting administrative evaluations to essential tasks, such as researcher recruitment and project funding, while establishing procedures grounded in principles of fairness.