DialDefer: A Framework for Detecting and Mitigating LLM Dialogic Deference

TL;DR

This paper introduces DialDefer, a framework to detect and reduce framing-induced judgment shifts in LLMs when evaluating dialogue, revealing significant variability depending on how prompts are framed.

Contribution

The paper presents DialDefer and the Dialogic Deference Score to quantify and mitigate framing effects in LLM judgments across multiple domains and models.

Findings

Framing causes large shifts in LLM judgments (up to 87pp) while accuracy remains stable.

Human-vs-LLM attribution significantly impacts judgment shifts.

Mitigation reduces deference but may lead to over-correction into skepticism.

Abstract

LLMs are increasingly used as third-party judges, yet their reliability when evaluating speakers in dialogue remains poorly understood. We show that LLMs judge identical claims differently depending on framing: the same content elicits different verdicts when presented as a statement to verify ("Is this statement correct?") versus attributed to a speaker ("Is this speaker correct?"). We call this dialogic deference and introduce DialDefer, a framework for detecting and mitigating these framing-induced judgment shifts. Our Dialogic Deference Score (DDS) captures directional shifts that aggregate accuracy obscures. Across nine domains, 3k+ instances, and four models, conversational framing induces large shifts (|DDS| up to 87pp, p < .0001) while accuracy remains stable (<2pp), with effects amplifying 2-4x on naturalistic Reddit conversations. Models can shift toward agreement (deference) or disagreement (skepticism) depending on domain -- the same model ranges from DDS = -53 on graduate-level science to +58 on social judgment. Ablations reveal that human-vs-LLM attribution drives the largest shifts (17.7pp swing), suggesting models treat disagreement with humans as more costly than with AI. Mitigation attempts reduce deference but can over-correct into skepticism, framing this as a calibration problem beyond accuracy optimization.