The first Italian research assessment exercise: a bibliometric perspective

TL;DR

This paper analyzes Italy's first national research assessment, comparing peer review and bibliometric indicators to evaluate research quality and propose a cost-effective, integrated evaluation method for future assessments.

Contribution

It provides a large-scale comparison of peer review and bibliometric indicators, demonstrating their relationship and advocating for combined use in future research evaluations.

Findings

Bibliometric indicators correlate with peer review judgments.

The assessment system favors merit-based evaluation over compliance.

Proposed integrated evaluation can reduce costs and improve accuracy.

Abstract

In December 2003, seventeen years after the first UK research assessment exercise, Italy started up its first-ever national research evaluation, with the aim to evaluate, using the peer review method, the excellence of the national research production. The evaluation involved 20 disciplinary areas, 102 research structures, 18,500 research products and 6,661 peer reviewers (1,465 from abroad); it had a direct cost of 3.55 millions Euros and a time length spanning over 18 months. The introduction of ratings based on ex post quality of output and not on ex ante respect for parameters and compliance is an important leap forward of the national research evaluation system toward meritocracy. From the bibliometric perspective, the national assessment offered the unprecedented opportunity to perform a large-scale comparison of peer review and bibliometric indicators for an important share of the Italian research production. The present investigation takes full advantage of this opportunity to test whether peer review judgements and (article and journal) bibliometric indicators are independent variables and, in the negative case, to measure the sign and strength of the association. Outcomes allow us to advocate the use of bibliometric evaluation, suitably integrated with expert review, for the forthcoming national assessment exercises, with the goal of shifting from the assessment of research excellence to the evaluation of average research performance without significant increase of expenses.