Measures and Mismeasures of Scientific Quality

TL;DR

This paper introduces a Bayesian approach to evaluate scientific quality measures, revealing that common metrics like 'papers per year' and 'h-index' are unreliable, while citation counts are more accurate for predicting future performance.

Contribution

The paper develops a Bayesian framework for assessing the reliability of scientific quality metrics and compares their effectiveness using citation data.

Findings

'Papers per year' and 'h-index' lack accuracy and precision.

Citation counts are reliable for predicting future performance.

Reliable predictions can be made with as few as 50 publications.

Abstract

We present a general Bayesian method for quantifying the statistical reliability of one-dimensional measures of scientific quality based on citation data. Two quality measures used in practice -- ``papers per year'' and ``Hirsch's $h$'' -- are shown to lack the accuracy and precision necessary to be useful. The mean, median and maximum number of citations are on the other hand reliable and permit accurate predictions of future author performance on the basis of as few as 50 publications.