Measuring Metrics - A forty year longitudinal cross-validation of citations, downloads, and peer review in Astrophysics

TL;DR

This study evaluates the effectiveness of citation and download metrics in predicting scholarly performance over four decades in astrophysics, revealing their capabilities and limitations through longitudinal analysis.

Contribution

It provides a forty-year longitudinal validation of citation, download, and peer review measures in assessing individual scholarly impact in astrophysics.

Findings

Lognormal distribution of measures implies multiplicative uncertainties.

A log difference of 0.3 dex is needed for 95% confidence in comparing individuals.

Metrics have inherent limitations in predicting future performance.

Abstract

Citation measures, and newer altmetric measures such as downloads are now commonly used to inform personnel decisions. How well do or can these measures measure or predict the past, current of future scholarly performance of an individual? Using data from the Smithsonian/NASA Astrophysics Data System we analyze the publication, citation, download, and distinction histories of a cohort of 922 individuals who received a U.S. PhD in astronomy in the period 1972-1976. By examining the same and different measures at the same and different times for the same individuals we are able to show the capabilities and limitations of each measure. Because the distributions are lognormal measurement uncertainties are multiplicative; we show that in order to state with 95% confidence that one person's citations and/or downloads are significantly higher than another person's, the log difference in the ratio of counts must be at least 0.3 dex, which corresponds to a multiplicative factor of two.