Web of Science: showing a bug today that can mislead scientific research output's prediction

TL;DR

This paper reveals a subtle flaw in the Web of Science database that can inflate research output metrics, potentially misleading evaluations of scientific achievement and impacting decisions based on these metrics.

Contribution

It identifies and demonstrates a specific bug in Web of Science that affects the accuracy of research output measurements used in scientific evaluation.

Findings

A software analysis of over 100,000 articles uncovered the bug.

The flaw can artificially inflate research output metrics.

This issue compromises the reliability of the h index and related evaluations.

Abstract

As it happened in all domains of human activities, economic issues and the increase of people working in scientific research have altered the way scientific production is evaluated so as the objectives of performing the evaluation. Introduced in 2005 by J. E. Hirsch as an indicator able to measure individual scientific output not only in terms of quantity, but also in terms of quality, h index has spread throughout the world. In 2007, Hirsch proposed its adoption also as the best to predict future scientific achievement and, consequently, a useful guide for investments in research and for institutions when hiring members for their scientific staff. Since then, several authors have also been using the Thomson ISI Web of Science database to develop their proposals for evaluating research output. Here, using a software we have developed, we analyse more than 100 thousand articles and show that a subtle flaw in Web of Science can inflate the results of info collected, therefore compromising the exactness and, consequently, the effectiveness of Hirsch's proposal and its variations.