Demanding peer review is associated with higher impact in published science

TL;DR

This study uses a large language model pipeline to analyze peer review correspondence, revealing that more intense review and criticism are linked to higher citation impact and that review styles vary across disciplines.

Contribution

It introduces a novel method to quantify peer review dynamics at scale and uncovers how review intensity correlates with scientific impact and disciplinary differences.

Findings

Review pressure is concentrated in the first review round.

Higher criticism and revision burden are associated with higher later citation impact.

Review styles differ more across fields than in review length.

Abstract

Peer review shapes which scientific claims enter the published record, but its internal dynamics are hard to measure at scale because reviewer criticism and author revision are usually embedded in long, unstructured correspondence. Here we use a fixed-prompt large language model pipeline to convert the review correspondence of \textit{Nature Communications} papers published from 2017 to 2024 into structured reviewer--author interactions. We find that review pressure is concentrated in the first round and focused disproportionately on core claims rather than peripheral presentation. Higher average opinion strength is also associated with more reviewer disagreement, while review patterns vary little with broad team attributes, consistent with relatively impartial evaluation. Contrary to the intuition that stronger papers should pass review more smoothly, with greater reviewer--author agreement and less extensive revision, we find that stronger criticism, higher-quality comments, and greater revision burden are associated with higher later citation impact within accepted papers. We finally show that fields differ more in review style than in review length, pointing to disciplinary variation in how criticism is negotiated and resolved. These findings position open peer review not just as a gatekeeping mechanism but as a measurable record of how influential scientific claims are challenged, defended, and revised before entering the published record.