Counterfactual LLM-based Framework for Measuring Rhetorical Style

TL;DR

This paper introduces a novel LLM-based counterfactual framework to quantify rhetorical style in scientific papers, revealing its impact on attention and the influence of LLM writing assistance.

Contribution

It presents a new method to measure rhetorical style independently of content using multiple LLM personas and pairwise evaluations, validated on thousands of ML papers.

Findings

Rhetorical strength predicts citations and media attention.

Rhetorical style increased sharply after 2023.

LLM-based writing aid largely drives the rise in rhetorical strength.

Abstract

The rise of AI has fueled growing concerns about ``hype'' in machine learning papers, yet a reliable way to quantify rhetorical style independently of substantive content has remained elusive. Because bold language can stem from either strong empirical results or mere rhetorical style, it is often difficult to distinguish between the two. To disentangle rhetorical style from substantive content, we introduce a counterfactual, LLM-based framework: multiple LLM rhetorical personas generate counterfactual writings from the same substantive content, an LLM judge compares them through pairwise evaluations, and the outcomes are aggregated using a Bradley--Terry model. Applying this method to 8,485 ICLR submissions sampled from 2017 to 2025, we generate more than 250,000 counterfactual writings and provide a large-scale quantification of rhetorical style in ML papers. We find that visionary framing significantly predicts downstream attention, including citations and media attention, even after controlling for peer-review evaluations. We also observe a sharp rise in rhetorical strength after 2023, and provide empirical evidence showing that this increase is largely driven by the adoption of LLM-based writing assistance. The reliability of our framework is validated by its robustness to the choice of personas and the high correlation between LLM judgments and human annotations. Our work demonstrates that LLMs can serve as instruments to measure and improve scientific evaluation.