How reliable are unsupervised author disambiguation algorithms in the assessment of research organization performance?

TL;DR

This paper investigates the bias introduced by unsupervised author disambiguation algorithms in research performance rankings of organizations, comparing them to supervised methods to assess their reliability and implications for policy decisions.

Contribution

It provides a comparative analysis of unsupervised versus supervised author disambiguation algorithms in research evaluation, highlighting biases and proposing a methodology for assessment.

Findings

Unsupervised algorithms can introduce significant biases in research rankings.

Supervised algorithms with input data yield more accurate organization performance assessments.

The developed methodology can be applied for broader comparative analyses in research evaluation.

Abstract

The paper examines extent of bias in the performance rankings of research organisations when the assessments are based on unsupervised author-name disambiguation algorithms. It compares the outcomes of a research performance evaluation exercise of Italian universities using the unsupervised approach by Caron and van Eck (2014) for derivation of the universities' research staff, with those of a benchmark using the supervised algorithm of D'Angelo, Giuffrida, and Abramo (2011), which avails of input data. The methodology developed could be replicated for comparative analyses in other frameworks of national or international interest, meaning that practitioners would have a precise measure of the extent of distortions inherent in any evaluation exercises using unsupervised algorithms. This could in turn be useful in informing policy-makers' decisions on whether to invest in building national research staff databases, instead of settling for the unsupervised approaches with their measurement biases.