Pressure, What Pressure? Sycophancy Disentanglement in Language Models via Reward Decomposition

TL;DR

This paper introduces a reward decomposition method to reduce sycophancy in large language models by disentangling pressure effects from evidence-based responses, leading to more truthful outputs.

Contribution

It proposes the first reward decomposition approach, GRPO, to disentangle pressure and evidence effects, improving model resistance to social pressure in responses.

Findings

Consistently reduces sycophancy across five models and metrics.

Decomposition terms independently govern specific behavioral dimensions.

Resistance to pressure generalizes beyond training conditions, reducing answer priming by up to 17 points.

Abstract

Large language models exhibit sycophancy, the tendency to shift their stated positions toward perceived user preferences or authority cues regardless of evidence. Standard alignment methods fail to correct this because scalar reward models conflate two distinct failure modes into a single signal: pressure capitulation, where the model changes a correct answer under social pressure, and evidence blindness, where the model ignores the provided context entirely. We operationalise sycophancy through formal definitions of pressure independence and evidence responsiveness, serving as a working framework for disentangled training rather than a definitive characterisation of the phenomenon. We propose the first approach to sycophancy reduction via reward decomposition, introducing a multi-component Group Relative Policy Optimisation (GRPO) reward that decomposes the training signal into five terms: pressure resistance, context fidelity, position consistency, agreement suppression, and factual correctness. We train using a contrastive dataset pairing pressure-free baselines with pressured variants across three authority levels and two opposing evidence contexts. Across five base models, our two-phase pipeline consistently reduces sycophancy on all metric axes, with ablations confirming that each reward term governs an independent behavioural dimension. The learned resistance to pressure generalises beyond our training methodology and prompt structure, reducing answer-priming sycophancy by up to 17 points on SycophancyEval despite the absence of such pressure forms during training.