Is the panel fair? Evaluating panel compositions through network analysis. The case of research assessments in Italy

TL;DR

This paper proposes an empirical network analysis method to evaluate the fairness of panel compositions in research assessments, revealing potential biases in panels appointed by authorities compared to randomly selected panels.

Contribution

It introduces a novel approach using co-authorship, publication, and affiliation networks to assess panel fairness in research evaluations.

Findings

Panels appointed by authorities had higher interconnectedness than the control panel.

Connectedness correlates with potential biases in panel composition.

The method provides a new way to evaluate fairness in research assessment panels.

Abstract

Research evaluation is usually governed by panels of peers. Procedural fairness refers to the principles that ensures decisions are made through a fair and transparent process. It requires that the composition of panels is fair. A fair panel is usually defined in terms of observable characteristics of scholars such as gender or affiliations. The formal adherence to these criteria is not sufficient to guarantee a fair composition in terms of scholarly thinking, background, or policy orientation. An empirical strategy for exploring the fairness in the intellectual composition of panels is proposed, based on the observation of links between panellists. The case study regards the three panels selected to evaluate research in economics, statistics and business during the Italian research assessment exercises. The first two panels were appointed directly by the governmental agency responsible for the evaluation, while the third was randomly selected. Hence the third panel can be considered as a control for evaluating about the fairness of the others. The fair representation is explored by comparing the networks of panellists based on their co-authorship relations, the networks based on journals in which they published and the networks based on their affiliated institutions (universities, research centres and newspapers). The results show that the members of the first two panels had connections much higher than the members of the control group. Hence the composition of the first two panels should be considered as unfair, as the results of the research assessments.