Scientific impact quantity and quality: Analysis of two sources of bibliographic data

TL;DR

This paper compares bibliometric data from Thomson/ISI and Google Scholar, analyzing their agreement and data quality to improve impact measurement of scientific publications.

Contribution

It provides a comparative analysis of two major bibliographic data sources, highlighting their agreement and potential complementarities for impact assessment.

Findings

Good agreement between Thomson/ISI and Google Scholar citation counts

Data quality varies across sources but shows promising complementarities

Multiple data sources enhance robustness of impact measurements

Abstract

Attempts to understand the consequence of any individual scientist's activity within the long-term trajectory of science is one of the most difficult questions within the philosophy of science. Because scientific publications play such as central role in the modern enterprise of science, bibliometric techniques which measure the ``impact'' of an individual publication as a function of the number of citations it receives from subsequent authors have provided some of the most useful empirical data on this question. Until recently, Thompson/ISI has provided the only source of large-scale ``inverted'' bibliographic data of the sort required for impact analysis. In the end of 2004, Google introduced a new service, GoogleScholar, making much of this same data available. Here we analyze 203 publications, collectively cited by more than 4000 other publications. We show surprisingly good agreement between data citation counts provided by the two services. Data quality across the systems is analyzed, and potentially useful complementarities between are considered. The additional robustness offered by multiple sources of such data promises to increase the utility of these measurements as open citation protocols and open access increase their impact on electronic scientific publication practices.