Quantifying perceived impact of scientific publications

TL;DR

This study empirically examines whether citation counts accurately reflect scientific impact as perceived by experts, revealing biases and the conditions under which citations align with expert opinions.

Contribution

It provides the first large-scale empirical verification of the relationship between citation counts and perceived impact, highlighting psychological biases influencing expert assessments.

Findings

Researchers prefer their own papers over highly cited ones.

Weak correlation between citation counts and expert preferences across different papers.

Bias is reduced when experts evaluate their own papers, aligning impact perception with citation data.

Abstract

Citations are commonly held to represent scientific impact. To date, however, there is no empirical evidence in support of this postulate that is central to research assessment exercises and Science of Science studies. Here, we report on the first empirical verification of the degree to which citation numbers represent scientific impact as it is actually perceived by experts in their respective field. We run a large-scale survey of about 2000 corresponding authors who performed a pairwise impact assessment task across more than 20000 scientific articles. Results of the survey show that citation data and perceived impact do not align well, unless one properly accounts for strong psychological biases that affect the opinions of experts with respect to their own papers vs. those of others. First, researchers tend to largely prefer their own publications to the most cited papers in their field of research. Second, there is only a mild positive correlation between the number of citations of top-cited papers in given research areas and expert preference in pairwise comparisons. This also applies to pairs of papers with several orders of magnitude differences in their total number of accumulated citations. However, when researchers were asked to choose among pairs of their own papers, thus eliminating the bias favouring one's own papers over those of others, they did systematically prefer the most cited article. We conclude that, when scientists have full information and are making unbiased choices, expert opinion on impact is congruent with citation numbers.