Do open citations give insights on the qualitative peer-review evaluation in research assessments? An analysis of the Italian National Scientific Qualification

TL;DR

This study investigates whether open citation data can support peer review in research assessments, specifically in the Italian National Scientific Qualification, by analyzing citation metrics and machine learning models to replicate committee decisions.

Contribution

It demonstrates that citation-based metrics derived from open data can provide insights into peer review processes, especially for non-citation disciplines, and highlights the importance of citation relationships in evaluations.

Findings

Citation relationships influence peer-review decisions in non-citation disciplines.

Open citation data can support and possibly enhance qualitative research assessments.

Machine learning models can replicate NSQ committee decisions using citation metrics.

Abstract

In the past, several works have investigated ways for combining quantitative and qualitative methods in research assessment exercises. Indeed, the Italian National Scientific Qualification (NSQ), i.e. the national assessment exercise which aims at deciding whether a scholar can apply to professorial academic positions as Associate Professor and Full Professor, adopts a quantitative and qualitative evaluation process: it makes use of bibliometrics followed by a peer-review process of candidates' CVs. The NSQ divides academic disciplines into two categories, i.e. citation-based disciplines (CDs) and non-citation-based disciplines (NDs), a division that affects the metrics used for assessing the candidates of that discipline in the first part of the process, which is based on bibliometrics. In this work, we aim at exploring whether citation-based metrics, calculated only considering open bibliographic and citation data, can support the human peer-review of NDs and yield insights on how it is conducted. To understand if and what citation-based (and, possibly, other) metrics provide relevant information, we created a series of machine learning models to replicate the decisions of the NSQ committees. As one of the main outcomes of our study, we noticed that the strength of the citational relationship between the candidate and the commission in charge of assessing his/her CV seems to play a role in the peer-review phase of the NSQ of NDs.