GRASP: Deterministic argument ranking in interaction graphs

TL;DR

GRASP is a deterministic framework that improves the consistency and transparency of argument ranking in interaction graphs by aggregating local judgments into a global, stable ranking, addressing issues of model disagreement.

Contribution

It introduces GRASP, a novel attack-defense propagation method that yields more reproducible and transparent argument rankings compared to holistic LLM judgments.

Findings

Local interaction judgments are more reproducible than holistic rankings.

GRASP produces more consistent global rankings than holistic LLM judgments.

GRASP scores measure structural sufficiency, not persuasion or factuality.

Abstract

Large language models are increasingly deployed as automated judges to evaluate the strength of arguments. As this role expands, their legitimacy depends on consistency, transparency, and the ability to separate argumentative structure from rhetorical appeal. However, we show that holistic judging - a common LLM-as-a-Judge practice where a model provides a global verdict on a debate - suffers from substantial inter-model disagreement. We argue that this instability arises from collapsing a debate's complex interaction structure into a single opaque score. To address this, we propose GRASP (Gradual Ranking with Attacks and Support Propagation), a deterministic framework that aggregates stable local interaction judgments into a global ranking via a convergent attack--defense propagation operator. We show that local interaction judgments are more reproducible than holistic rankings in LLM-as-a-Judge evaluations, allowing GRASP to produce more consistent global rankings. We further show that GRASP scores do not correlate with human "convincingness" labels, highlighting a vital sociotechnical distinction: GRASP does not measure persuasion, factuality, or rhetorical appeal, but structural sufficiency - a defense-aware notion of argument robustness over the explicit interaction graph. Overall, GRASP offers a transparent and auditable alternative to holistic LLM judging.