General Preference Reinforcement Learning

TL;DR

This paper introduces General Preference Reinforcement Learning (GPRL), a novel method that uses a structured multi-dimensional preference model to improve open-ended language model alignment and robustness.

Contribution

The paper proposes GPRL, which leverages a multi-dimensional preference embedding and a drift monitor to enhance policy updates and prevent reward hacking in language models.

Findings

GPRL achieves a 56.51% win rate on AlpacaEval 2.0.

GPRL outperforms SimPO and SPPO on Arena-Hard, MT-Bench, and WildBench.

GPRL resists reward hacking across extended training.

Abstract

Post-training has split large language model (LLM) alignment into two largely disconnected tracks. Online reinforcement learning (RL) with verifiable rewards drives emergent reasoning on math and code but depends on a programmatic verifier that cannot reach open-ended tasks, while preference optimization handles open-ended generation yet forgoes the continuous exploration that powers online RL. Closing this gap requires a verifier for open-ended quality, but a scalar reward model is the wrong shape for the job. Quality is multi-dimensional, and any scalar score is an incomplete proxy that lets online RL collapse onto whichever axis the score is most sensitive to. We turn instead to the General Preference Model (GPM), which embeds responses into $k$ skew-symmetric subspaces and represents preference as a structured, intransitivity-aware comparison. Building on this, we propose General Preference Reinforcement Learning (GPRL), which carries the $k$-way structure through to the policy update. GPRL computes per-dimension group-relative advantages, normalizes each on its own scale so no axis can dominate, and aggregates them with context-dependent eigenvalues. The same structure powers a closed-loop drift monitor that detects single-axis exploitation and corrects it on the fly by reweighting dimensions and tightening the trust region. Starting from $\texttt{Llama-3-8B-Instruct}$, GPRL reaches a length-controlled win rate of $56.51\%$ on AlpacaEval~2.0 while also outperforming SimPO and SPPO on Arena-Hard, MT-Bench, and WildBench by resisting reward hacking across extended training runs.