Explaining and Preventing Alignment Collapse in Iterative RLHF

TL;DR

This paper analyzes the feedback loop in iterative RLHF, identifies the cause of alignment collapse, and proposes FPO to mitigate it, improving alignment in language model training.

Contribution

It introduces a formal analysis of the feedback dynamics in RLHF and proposes FPO, a novel method to prevent alignment collapse during iterative training.

Findings

Standard RLHF suffers from alignment collapse due to ignored policy steering effects.

FPO effectively prevents alignment collapse in controlled environments.

FPO improves alignment in Llama-3.2-1B language model pipeline.

Abstract

Reinforcement learning from human feedback (RLHF) typically assumes a static or non-strategic reward model (RM). In iterative deployment, however, the policy generates the data on which the RM is retrained, creating a feedback loop. Building on the Stackelberg game formulation of this interaction, we derive an analytical decomposition of the policy's true optimization gradient into a standard policy gradient and a parameter-steering term that captures the policy's influence on the RM's future parameters. We show that standard iterative RLHF, which drops this steering term entirely, suffers from alignment collapse: the policy systematically exploits the RM's blind spots, producing low-quality, high-reward outputs whose feedback reinforces the very errors it exploits. To mitigate this, we propose foresighted policy optimization (FPO), a mechanism-design intervention that restores the missing steering term by regularizing the policy's parameter-steering effect on RM updates. We instantiate FPO via a scalable first-order approximation and demonstrate that it prevents alignment collapse on both controlled environments and an LLM alignment pipeline using Llama-3.2-1B.