Why Is RLHF Alignment Shallow? A Gradient Analysis

TL;DR

This paper analyzes why reinforcement learning with human feedback (RLHF) produces shallow alignment in large language models, revealing that gradient signals focus only on early harm-determining tokens and proposing a new objective to address this limitation.

Contribution

It provides a theoretical gradient analysis explaining the shallow nature of RLHF alignment and introduces a new objective to generate deeper alignment signals.

Findings

Gradient signals concentrate on early tokens where harm is decided.

Standard alignment objectives cannot produce deep alignment.

A new recovery penalty objective creates gradient signals at all positions.

Abstract

Why is safety alignment in LLMs shallow? We prove that gradient-based alignment inherently concentrates on positions where harm is decided and vanishes beyond. Using a martingale decomposition of sequence-level harm, we derive an exact characterization of alignment gradients. The gradient at position $t$ equals the covariance between the conditional expected harm and the score function. This implies that positions beyond the harm horizon where the output's harmfulness is already determined receive zero gradient signal during training. This explains empirical observations that KL divergence between aligned and base models concentrates on early tokens. Consequently, standard alignment objectives cannot produce deep alignment, regardless of optimization quality. We introduce the concept of harm information $I_t$, which quantifies each position's influence on harm, and prove that equilibrium KL divergence tracks this quantity. Finally, we derive an objective based on recovery penalties that creates gradient signal at all positions, providing theoretical grounding for empirically successful data augmentation techniques.