Rewards as Labels: Revisiting RLVR from a Classification Perspective

TL;DR

This paper introduces Rewards as Labels (REAL), a classification-based framework for reinforcement learning with verifiable rewards, improving training stability and performance over existing methods like GRPO.

Contribution

REAL reformulates policy optimization as a classification problem using rewards as categorical labels, addressing gradient misassignment and domination issues in RLVR.

Findings

REAL outperforms GRPO and variants on reasoning benchmarks.

REAL improves Pass@1 by 6.7% on 1.5B models.

Even with simple binary cross-entropy, REAL surpasses DAPO.

Abstract

Reinforcement Learning with Verifiable Rewards has recently advanced the capabilities of Large Language Models in complex reasoning tasks by providing explicit rule-based supervision. Among RLVR methods, GRPO and its variants have achieved strong empirical performance. Despite their success, we identify that they suffer from Gradient Misassignment in Positives and Gradient Domination in Negatives, which lead to inefficient and suboptimal policy updates. To address these issues, we propose Rewards as Labels (REAL), a novel framework that revisits verifiable rewards as categorical labels rather than scalar weights, thereby reformulating policy optimization as a classification problem. Building on this, we further introduce anchor logits to enhance policy learning. Our analysis reveals that REAL induces a monotonic and bounded gradient weighting, enabling balanced gradient allocation across rollouts and effectively mitigating the identified mismatches. Extensive experiments on mathematical reasoning benchmarks show that REAL improves training stability and consistently outperforms GRPO and strong variants such as DAPO. On the 1.5B model, REAL improves average Pass@1 over DAPO by 6.7%. These gains further scale to 7B model, REAL continues to outperform DAPO and GSPO by 6.2% and 1.7%, respectively. Notably, even with a vanilla binary cross-entropy, REAL remains stable and exceeds DAPO by 4.5% on average.