The Noisy Path from Source to Citation: Measuring How Scholars Engage with Past Research

TL;DR

This paper introduces a computational method to measure how accurately citations reflect original claims, revealing systematic differences based on factors like recency, accessibility, and author characteristics, and highlighting limitations of citation counts.

Contribution

It presents a scalable pipeline to quantify citation fidelity at sentence level across a large dataset, uncovering factors influencing citation accuracy and the 'telephone effect' in scholarly communication.

Findings

Citation fidelity is higher for recent, accessible papers.

Authors with lower H-index and medium-sized teams cite more faithfully.

Low fidelity citations tend to propagate errors through subsequent papers.

Abstract

Academic citations are widely used for evaluating research and tracing knowledge flows. Such uses typically rely on raw citation counts and neglect variability in citation types. In particular, citations can vary in their fidelity as original knowledge from cited studies may be paraphrased, summarized, or reinterpreted, possibly wrongly, leading to variation in how much information changes from cited to citing paper. In this study, we introduce a computational pipeline to quantify citation fidelity at scale. Using full texts of papers, the pipeline identifies citations in citing papers and the corresponding claims in cited papers, and applies supervised models to measure fidelity at the sentence level. Analyzing a large-scale multi-disciplinary dataset of approximately 13 million citation sentence pairs, we find that citation fidelity is higher when authors cite papers that are 1) more recent and intellectually close, 2) more accessible, and 3) the first author has a lower H-index and the author team is medium-sized. Using a quasi-experiment, we establish the "telephone effect" - when citing papers have low fidelity to the original claim, future papers that cite the citing paper and the original have lower fidelity to the original. Our work reveals systematic differences in citation fidelity, underscoring the limitations of analyses that rely on citation quantity alone and the potential for distortion of evidence.