Benchmarking some Portuguese S&T system research units: 2nd Edition

TL;DR

This paper conducts a benchmarking study of Portuguese research units using bibliometric and productivity metrics from 2003-2006, demonstrating the utility of automated tools for objective evaluation.

Contribution

It introduces a methodology for automatic, objective benchmarking of research units based on publicly available data, enhancing evaluation efficiency.

Findings

Research units show varied productivity and impact levels.

Automated tools effectively reduce evaluation costs.

Benchmarking can support peer review processes.

Abstract

The increasing use of productivity and impact metrics for evaluation and comparison, not only of individual researchers but also of institutions, universities and even countries, has prompted the development of bibliometrics. Currently, metrics are becoming widely accepted as an easy and balanced way to assist the peer review and evaluation of scientists and/or research units, provided they have adequate precision and recall. This paper presents a benchmarking study of a selected list of representative Portuguese research units, based on a fairly complete set of parameters: bibliometric parameters, number of competitive projects and number of PhDs produced. The study aimed at collecting productivity and impact data from the selected research units in comparable conditions i.e., using objective metrics based on public information, retrievable on-line and/or from official sources and thus verifiable and repeatable. The study has thus focused on the activity of the 2003-06 period, where such data was available from the latest official evaluation. The main advantage of our study was the application of automatic tools, achieving relevant results at a reduced cost. Moreover, the results over the selected units suggest that this kind of analyses will be very useful to benchmark scientific productivity and impact, and assist peer review.