Confidence intervals for normalised citation counts: Can they delimit underlying research capability?

TL;DR

This study examines whether confidence intervals for normalized citation counts can reliably delineate a research group's underlying impact potential, finding that most future impact measures fall within previous confidence intervals, suggesting they may be useful indicators.

Contribution

The paper introduces the concept of research capability and empirically tests if confidence intervals can effectively delimit this capability in citation impact assessments.

Findings

Approximately 95% of future citation impact values fall within previous confidence intervals.

The decline over time suggests confidence intervals may approximate underlying research capability.

Results are consistent regardless of inclusion of internationally collaborative articles.

Abstract

Normalised citation counts are routinely used to assess the average impact of research groups or nations. There is controversy over whether confidence intervals for them are theoretically valid or practically useful. In response, this article introduces the concept of a group's underlying research capability to produce impactful research. It then investigates whether confidence intervals could delimit the underlying capability of a group in practice. From 123120 confidence interval comparisons for the average citation impact of the national outputs of ten countries within 36 individual large monodisciplinary journals, moderately fewer than 95% of subsequent indicator values fall within 95% confidence intervals from prior years, with the percentage declining over time. This is consistent with confidence intervals effectively delimiting the research capability of a group, although it does not prove that this is the cause of the results. The results are unaffected by whether internationally collaborative articles are included.