Blockwise Advantage Estimation for Multi-Objective RL with Verifiable Rewards

TL;DR

This paper introduces Blockwise Advantage Estimation, a method for multi-objective reinforcement learning that assigns advantages to specific text blocks, reducing reward interference and improving structured generation tasks.

Contribution

It proposes a novel advantage estimation technique that assigns separate advantages to each objective within text blocks, enabling better multi-objective optimization without nested rollouts.

Findings

Mitigates reward interference in structured generation tasks

Achieves competitive performance with reward-designed approaches

Preserves test-time gains from confidence-weighted ensembling

Abstract

Group Relative Policy Optimization (GRPO) assigns a single scalar advantage to all tokens in a completion. For structured generations with explicit segments and objectives, this couples unrelated reward signals across segments, leading to objective interference and misattributed credit. We propose Blockwise Advantage Estimation, a family of GRPO-compatible methods that assigns each objective its own advantage and applies it only to the tokens in the corresponding text block, reducing reliance on hand-designed scalar rewards and scaling naturally to additional objectives. A key challenge is estimating advantages for later blocks whose rewards are conditioned on sampled prefixes; standard unbiased approaches require expensive nested rollouts from intermediate states. Concretely, we introduce an Outcome-Conditioned Baseline that approximates intermediate state values using only within-group statistics by stratifying samples according to a prefix-derived intermediate outcome. On math tasks with uncertainty estimation, our method mitigates reward interference, is competitive with a state-of-the-art reward-designed approach, and preserves test-time gains from confidence-weighted ensembling. More broadly, it provides a modular recipe for optimizing sequential objectives in structured generations without additional rollouts.