Do Citations and Readership Identify Seminal Publications?

TL;DR

This study evaluates whether citation and Mendeley reader counts can identify seminal publications, finding citations outperform random chance while reader counts do not, based on a new dataset of influential and review papers.

Contribution

The paper introduces the TrueImpactDataset for validating research metrics and demonstrates that citation counts are somewhat effective in identifying influential research, unlike reader counts.

Findings

Citation counts outperform random baseline by 10%.

Mendeley reader counts do not outperform random baseline.

Citation counts achieve 63% accuracy in identifying influential papers.

Abstract

In this paper, we show that citation counts work better than a random baseline (by a margin of 10%) in distinguishing excellent research, while Mendeley reader counts don't work better than the baseline. Specifically, we study the potential of these metrics for distinguishing publications that caused a change in a research field from those that have not. The experiment has been conducted on a new dataset for bibliometric research called TrueImpactDataset. TrueImpactDataset is a collection of research publications of two types -- research papers which are considered seminal works in their area and papers which provide a literature review of a research area. We provide overview statistics of the dataset and propose to use it for validating research evaluation metrics. Using the dataset, we conduct a set of experiments to study how citation and reader counts perform in distinguishing these publication types, following the intuition that causing a change in a field signifies research contribution. We show that citation counts help in distinguishing research that strongly influenced later developments from works that predominantly discuss the current state of the art with a degree of accuracy (63%, i.e. 10% over the random baseline). In all setups, Mendeley reader counts perform worse than a random baseline.