Bringing Value Models Back: Generative Critics for Value Modeling in LLM Reinforcement Learning

TL;DR

This paper introduces Generative Actor-Critic (GenAC), a novel approach that uses generative critics with chain-of-thought reasoning to improve value modeling and credit assignment in LLM reinforcement learning.

Contribution

The paper proposes GenAC, replacing traditional value prediction with a generative critic, enhancing value approximation and out-of-distribution generalization in LLM RL.

Findings

GenAC improves value approximation and ranking reliability.

GenAC outperforms value-based and value-free baselines in downstream RL tasks.

In-Context Conditioning helps maintain critic calibration during training.

Abstract

Credit assignment is a central challenge in reinforcement learning (RL). Classical actor-critic methods address this challenge through fine-grained advantage estimation based on a learned value function. However, learned value models are often avoided in modern large language model (LLM) RL because conventional discriminative critics are difficult to train reliably. We revisit value modeling and argue that this difficulty is partly due to limited expressiveness. In particular, representation complexity theory suggests that value functions can be hard to approximate under the one-shot prediction paradigm used by existing value models, and our scaling experiments show that such critics do not improve reliably with scale. Motivated by this observation, we propose Generative Actor-Critic (GenAC), which replaces one-shot scalar value prediction with a generative critic that performs chain-of-thought reasoning before producing a value estimate. We further introduce In-Context Conditioning, which helps the critic remain calibrated to the current actor throughout training. GenAC improves value approximation, ranking reliability, and out-of-distribution generalization, and these gains translate into stronger downstream RL performance than both value-based and value-free baselines. Overall, our results suggest that stronger value modeling is a promising direction for improving credit assignment in LLM reinforcement learning.