Auditing automated research assessment: an interpretable machine learning approach to validate funding criteria

TL;DR

This study uses interpretable machine learning to evaluate the validity of official research assessment criteria, revealing that only a subset of criteria significantly influence funding decisions in Brazil.

Contribution

It introduces an empirical, data-driven approach to validate and analyze the actual impact of regulatory criteria on research funding outcomes.

Findings

High predictive accuracy (AUC 0.96) of models distinguishing grant levels.

Key features like bibliographic output and supervision are most influential.

Several regulated criteria show no detectable impact on classification results.

Abstract

This paper empirically examines the practical validity of the official evaluation criteria underpinning the Research Productivity (PQ) Grant framework, as governed by the Brazilian National Council for Scientific and Technological Development (CNPq). By operationalizing regulatory dimensions (including bibliographic output, human resource training, and scientific recognition) as measurable variables extracted from CVs and OpenAlex bibliometric data, we treat policy-defined indicators as testable hypotheses rather than a priori assumptions. Using a block-based adaptation of the Boruta feature selection algorithm across several machine learning classifiers, we evaluate the statistical contribution of each dimension in distinguishing grant levels, with a focus on identifying top-tier (Level 1A) researchers. Our models achieve high predictive performance, with mean AUC scores reaching 0.96, indicating that PQ levels carry a robust and structured statistical signal. However, explanatory power is heavily concentrated within a limited subset of features, specifically bibliographic production, graduate-level supervision and institutional management roles. Conversely, several criteria explicitly emphasized in the regulations demonstrated no detectable statistical contribution to classification outcomes. These findings reveal a potential misalignment between the formal regulatory framework and the effective signals driving evaluation outcomes, suggesting that the practical evaluative signal is substantially more compact than officially stated and providing evidence-based insights for the refinement and transparency of research assessment policies.