Assessing evaluation procedures for individual researchers: the case of the Italian National Scientific Qualification

TL;DR

This paper evaluates the Italian National Scientific Qualification process, highlighting issues with bibliometric use and report quality, and suggests improvements for future assessments of individual researchers.

Contribution

It provides an assessment of ASN's evaluation methodology and report quality, revealing inconsistencies and non-compliance with best bibliometric practices.

Findings

ASN's methodology is not fully aligned with best bibliometric practices.

Significant variability exists in the quality of evaluation reports.

Recommendations are made to improve evaluation consistency and report quality.

Abstract

The Italian National Scientific Qualification (ASN) was introduced as a prerequisite for applying for tenured associate or full professor positions at state-recognized universities. The ASN is meant to attest that an individual has reached a suitable level of scientific maturity to apply for professorship positions. A five member panel, appointed for each scientific discipline, is in charge of evaluating applicants by means of quantitative indicators of impact and productivity, and through an assessment of their research profile. Many concerns were raised on the appropriateness of the evaluation criteria, and in particular on the use of bibliometrics for the evaluation of individual researchers. Additional concerns were related to the perceived poor quality of the final evaluation reports. In this paper we assess the ASN in terms of appropriateness of the applied methodology, and the quality of the feedback provided to the applicants. We argue that the ASN is not fully compliant with the best practices for the use of bibliometric indicators for the evaluation of individual researchers; moreover, the quality of final reports varies considerably across the panels, suggesting that measures should be put in place to prevent sloppy practices in future ASN rounds.