URPO: A Unified Reward & Policy Optimization Framework for Large Language Models

TL;DR

URPO introduces a unified framework that combines reward modeling and policy optimization in a single model and training phase, improving alignment and performance of large language models.

Contribution

The paper proposes URPO, a novel unified training method that integrates reward and policy optimization, reducing complexity and enhancing model performance.

Findings

Outperforms baseline with separate reward model in instruction-following and reasoning tasks.

Boosts instruction-following score from 42.24 to 44.84 on AlpacaEval.

Achieves a RewardBench score of 85.15, surpassing dedicated reward models.

Abstract

Large-scale alignment pipelines typically pair a policy model with a separately trained reward model whose parameters remain frozen during reinforcement learning (RL). This separation creates a complex, resource-intensive pipeline and suffers from a performance ceiling due to a static reward signal. We propose a novel framework, Unified Reward & Policy Optimization (URPO), that unifies instruction-following ("player") and reward modeling ("referee") within a single model and a single training phase. Our method recasts all alignment data-including preference pairs, verifiable reasoning, and open-ended instructions-into a unified generative format optimized by a single Group-Relative Policy Optimization (GRPO) loop. This enables the model to learn from ground-truth preferences and verifiable logic while simultaneously generating its own rewards for open-ended tasks. Experiments on the Qwen2.5-7B model demonstrate URPO's superiority. Our unified model significantly outperforms a strong baseline using a separate generative reward model, boosting the instruction-following score on AlpacaEval from 42.24 to 44.84 and the composite reasoning average from 32.66 to 35.66. Furthermore, URPO cultivates a superior internal evaluator as a byproduct of training, achieving a RewardBench score of 85.15 and surpassing the dedicated reward model it replaces (83.55). By eliminating the need for a separate reward model and fostering a co-evolutionary dynamic between generation and evaluation, URPO presents a simpler, more efficient, and more effective path towards robustly aligned language models.