Academics evaluating academics: a methodology to inform the review process on top of open citations

TL;DR

This paper proposes a methodology using open citation data and machine learning to analyze and replicate peer-review decisions in research assessment exercises, aiming to enhance understanding of evaluation processes.

Contribution

It introduces a novel approach combining open citation data with machine learning to explore the insights provided by citation metrics in research assessments.

Findings

Machine learning models can replicate peer-review decisions

Open citation data can inform research assessment processes

Insights from metrics may complement traditional review methods

Abstract

In the past, several works have investigated ways for combining quantitative and qualitative methods in research assessment exercises. In this work, we aim at introducing a methodology to explore whether citation-based metrics, calculated only considering open bibliographic and citation data, can yield insights on how human peer-review of research assessment exercises is conducted. To understand if and what metrics provide relevant information, we propose to use a series of machine learning models to replicate the decisions of the committees of the research assessment exercises.