Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective

TL;DR

This paper introduces ConSPO, a contrastive sequence-level policy optimization framework for RLVR, addressing limitations of prior methods and improving reasoning performance in language models.

Contribution

ConSPO reformulates RLVR scoring to align with likelihoods and employs a contrastive, curriculum-guided approach for better credit assignment and model training.

Findings

ConSPO outperforms strong RLVR baselines on mathematical reasoning benchmarks.

Reformulation reveals limitations of likelihood-misaligned scoring and score-insensitive credit assignment.

Extensive evaluations demonstrate consistent improvements across models and datasets.

Abstract

RLVR has become a widely adopted paradigm for improving LLMs' reasoning capabilities, and GRPO is one of its most representative algorithms. In this paper, we first show that GRPO admits an equivalent discriminative reformulation as a weighted positive-negative score difference. Under this view, GRPO increases sequence-level scores of verified positive rollouts and decreases those of negative rollouts, where the scores are averages of clipped token-level importance sampling ratios. This reformulation reveals two structural limitations of GRPO: likelihood-misaligned scoring, where clipped ratio-based surrogate scores are optimized instead of generation likelihoods, and score-insensitive credit assignment, where rollout-level credit is assigned without accounting for relative score gaps between positive and negative rollouts in the same group. To address these limitations, we propose ConSPO, a framework for Contrastive Sequence-level Policy Optimization in RLVR. ConSPO replaces GRPO's clipped ratio-based scores with length-normalized sequence log-probabilities, aligning the optimized rollout scores with the likelihoods used in autoregressive generation. It then optimizes a group-wise InfoNCE-style objective that contrasts each positive rollout against negative distractors from the same group, enabling credit assignment to depend on their relative scores. This contrastive formulation amplifies updates for poorly separated positives while concentrating suppressive updates on high-scoring negatives. Moreover, ConSPO introduces a curriculum-scheduled margin, guiding optimization from coarse positive-negative ordering in early training toward stronger separation in later stages. Extensive evaluations across diverse backbone models, parameter scales, and training datasets show that ConSPO consistently outperforms several strong RLVR baselines on challenging mathematical reasoning benchmarks.