The origin, consequence, and visibility of criticism in science

TL;DR

This study analyzes the origins, effects, and visibility of critical peer reviews in science, revealing that while criticism targets high-impact papers, it has limited influence on citation trajectories and remains largely unnoticed within the scientific community.

Contribution

It provides the first large-scale empirical analysis of critical letters, highlighting their limited impact and visibility despite their importance in scientific discourse.

Findings

Critical letters target highly-cited, interdisciplinary, and novel papers.

Receiving a critical letter does not significantly affect a paper's citation trajectory.

Critical letters are rarely cited by distant research communities, indicating limited visibility.

Abstract

Critique between peers plays a vital role in the production of scientific knowledge. Yet, there is limited empirical evidence on the origins of criticism, its effects on the papers and individuals involved, and its visibility within the scientific literature. Here, we address these gaps through a data-driven analysis of papers that received substantiated and explicit written criticisms. Our analysis draws on data representing over 3,000 ``critical letters'' -- papers explicitly published to critique another -- from four high profile journals, with each letter linked to its target paper. We find that the papers receiving critical letters are disproportionately among the most highly-cited in their respective journal and, to a lesser extent, among the most interdisciplinary and novel. However, despite the theoretical importance of criticism in scientific progress, we observe no evidence that receiving a critical letter affects a paper's citation trajectory or the productivity and citation impact of its authors. One explanation for the limited consequence of critical letters is that they often go unnoticed. Indeed, we find that critical letters attract only a small fraction of the citations received by their targets, even years after publication. An analysis of topical similarity between criticized papers and their citing papers indicates that critical letters are primarily cited by researchers actively engaged in a similar field of study, whereas they are overlooked by more distant communities. Although criticism is celebrated as a cornerstone to science, our findings reveal that it is concentrated on high-impact papers, has minimal measurable consequences, and suffers from limited visibility. These results raise important questions about the role and value of critique in scientific practice.