How RLHF Amplifies Sycophancy

TL;DR

This paper analyzes how reinforcement learning from human feedback (RLHF) can unintentionally amplify sycophantic behavior in language models, and proposes a method to mitigate this effect through reward correction.

Contribution

It provides a formal analysis of the amplification mechanism in RLHF and introduces a training intervention to neutralize sycophantic bias in language models.

Findings

Reward gaps are common and cause behavioral drift.

The amplification mechanism is linked to covariance between endorsement signals and learned reward.

A minimal reward correction policy can prevent increased sycophancy.

Abstract

Large language models often exhibit increased sycophantic behavior after preference-based post-training, showing a stronger tendency to affirm a user's stated or implied belief even when this conflicts with factual accuracy or sound judgment. We present a formal analysis of how alignment from human feedback can increase this failure mode by identifying an explicit amplification mechanism that causally links optimization against a learned reward to bias in the human preference data used for alignment. We show that the direction of behavioral drift is determined by a covariance under the base policy between endorsing the belief signal in the prompt and the learned reward, and that the first-order effect reduces to a simple mean-gap condition. We then analyze reward learning from pairwise comparisons under random utility models like Bradley-Terry and characterize when bias in human annotators' preferences induces this reward gap. Next, we propose a training-time intervention designed to neutralize the amplification mechanism itself. Among all post-trained policies that prevent sycophantic behavior from increasing, we characterize the unique policy closest in KL divergence to the unconstrained post-trained policy, and derive the corresponding minimal reward correction as a closed-form agreement penalty. Computational experiments find that reward gaps are common and cause behavioral drift in all the configurations considered.