When Right Meets Wrong: Bilateral Context Conditioning with Reward-Confidence Correction for GRPO

TL;DR

This paper introduces Bilateral Context Conditioning and Reward-Confidence Correction to enhance Group Relative Policy Optimization, leveraging contrastive learning and dynamic advantage adjustment to improve reasoning model training.

Contribution

It presents a novel contrastive reformulation of GRPO and new mechanisms that enable cross-sample reasoning comparison without extra sampling or models.

Findings

Consistent improvements on mathematical reasoning benchmarks.

Effective stabilization of training with Reward-Confidence Correction.

Applicable to various GRPO variants.

Abstract

Group Relative Policy Optimization (GRPO) has emerged as an effective method for training reasoning models. While it computes advantages based on group mean, GRPO treats each output as an independent sample during the optimization and overlooks a vital structural signal: the natural contrast between correct and incorrect solutions within the same group, thus ignoring the rich, comparative data that could be leveraged by explicitly pitting successful reasoning traces against failed ones. To capitalize on this, we present a contrastive reformulation of GRPO, showing that the GRPO objective implicitly maximizes the margin between the policy ratios of correct and incorrect samples. Building on this insight, we propose Bilateral Context Conditioning (BICC), a mechanism that allows the model to cross-reference successful and failed reasoning traces during the optimization, enabling a direct information flow across samples. We further introduce Reward-Confidence Correction (RCC) to stabilize training by dynamically adjusts the advantage baseline in GRPO using reward-confidence covariance derived from the first-order approximation of the variance-minimizing estimator. Both mechanisms require no additional sampling or auxiliary models and can be adapted to all GRPO variants. Experiments on mathematical reasoning benchmarks demonstrate consistent improvements across comprehensive models and algorithms. Code is available at \href{https://github.com/Skylanding/BiCC}{https://github.com/Skylanding/BiCC}.