Errors and secret data in the Italian research assessment exercise. A comment to a reply

TL;DR

This paper critiques the data and methodology of a study on Italy's research assessment system, highlighting errors, lack of transparency, and reproducibility issues, and warns about the unreliability of research based on inaccessible data.

Contribution

It exposes critical flaws and transparency issues in a key study evaluating Italy's research assessment exercise, emphasizing the need for data disclosure and reproducibility.

Findings

Identified errors in the data used by the criticized study.

Highlighted issues with sample representativeness and unverifiable claims.

Warned about the unreliability of research based on undisclosed Italian assessment data.

Abstract

Italy adopted a performance-based system for funding universities that is centered on the results of a national research assessment exercise, realized by a governmental agency (ANVUR). ANVUR evaluated papers by using 'a dual system of evaluation', that is by informed peer review or by bibliometrics. In view of validating that system, ANVUR performed an experiment for estimating the agreement between informed review and bibliometrics. Ancaiani et al. (2015) presents the main results of the experiment. Baccini and De Nicolao (2017) documented in a letter, among other critical issues, that the statistical analysis was not realized on a random sample of articles. A reply to the letter has been published by Research Evaluation (Benedetto et al. 2017). This note highlights that in the reply there are (1) errors in data, (2) problems with 'representativeness' of the sample, (3) unverifiable claims about weights used for calculating kappas, (4) undisclosed averaging procedures; (5) a statement about 'same protocol in all areas' contradicted by official reports. Last but not least: the data used by the authors continue to be undisclosed. A general warning concludes: many recently published papers use data originating from Italian research assessment exercise. These data are not accessible to the scientific community and consequently these papers are not reproducible. They can be hardly considered as containing sound evidence at least until authors or ANVUR disclose the data necessary for replication.