On the Predictability of Utilizing Rank Percentile to Evaluate Scientific Impact

TL;DR

This paper introduces a new rank percentile metric for evaluating scientific impact, demonstrating its high predictability and stability over time, with simple models providing effective predictions.

Contribution

The paper proposes a novel rank percentile indicator for scholars and shows its high predictability and stability, emphasizing the advantages of simple linear models.

Findings

Publication percentile is highly stable over time.

Scholar percentile exhibits short-term stability and can be predicted with linear regression.

Simple models outperform more complex ones in predictability and interpretability.

Abstract

Bibliographic metrics are commonly utilized for evaluation purposes within academia, often in conjunction with other metrics. These metrics vary widely across fields and change with the seniority of the scholar; consequently, the only way to interpret these values is by comparison with other academics within the same field who are of similar seniority. Among the field- and time- normalized indicators, rank percentile has grown in popularity, and it is preferred over other types of indicators. In this paper, we propose and justify a novel rank percentile indicator for scholars. Furthermore, we emphasize on the time factor that is built into the rank percentile, and we demonstrate that the rank percentile is highly predictable. The publication percentile is highly stable over time, while the scholar percentile exhibits short-term stability and can be predicted via a simple linear regression model. More advanced models that utilize extensive lists of features offer slightly superior performance; however, the simplicity and interpretability of the simple model impose significant advantages over the additional complexity of other models.