Pretrain Value, Not Reward: Decoupled Value Policy Optimization

TL;DR

This paper proposes a novel reinforcement learning framework called Decoupled Value Policy Optimization (DVPO) that pretrains a global value model offline, simplifying and stabilizing RLHF by eliminating the need for online critic training.

Contribution

The paper introduces DVPO, a method that pretrains a universal value model offline to guide policy learning, reducing complexity and improving stability in RLHF.

Findings

DVPO matches or surpasses state-of-the-art RLHF methods.

Pretraining a global value model simplifies reinforcement learning.

The approach reduces critic drift and trajectory sampling issues.

Abstract

In this paper, we explore how directly pretraining a value model simplifies and stabilizes reinforcement learning from human feedback (RLHF). In reinforcement learning, value estimation is the key to policy optimization, distinct from reward supervision. The value function predicts the \emph{return-to-go} of a partial answer, that is, how promising the partial answer is if it were continued to completion. In RLHF, however, the standard pipeline first pretrains a reward model and then learns a value function online, even though no new reward signals are available once preference data is collected. This makes critic learning redundant, as the process of training a reward model and then deriving a value model is informationally equivalent to directly pretraining a value model. Importantly, this requires no additional supervision, and our value model is trained on exactly the same data used for reward modeling. Building on this insight, we introduce \emph{Decoupled Value Policy Optimization} (DVPO), a framework that pretrains a \emph{Global Value Model} (GVM) offline and freezes it as a universal critic for policy learning. The GVM provides stable, fine-grained credit assignment without critic drift or trajectory sampling. Experiments across MT-Bench, Alpaca-Eval, and Arena-Hard demonstrate that DVPO matches or surpasses state-of-the-art RLHF methods. These results highlight RLHF can be reframed as policy-only optimization guided by a single pretrained value model.