Leveraging Error Diversity in Group Rollouts for Reinforcement Learning

TL;DR

This paper introduces Error Diversity Advantage Shaping (EDAS), a simple technique that improves reinforcement learning from verifiable rewards by leveraging error diversity in group responses, leading to consistent performance gains.

Contribution

The paper proposes EDAS, a novel, model-agnostic method that enhances RLVR by modulating advantage signals based on intra-group error diversity, improving training outcomes.

Findings

EDAS improves performance across multiple models and benchmarks.

Error diversity within group responses predicts training success.

EDAS yields an average of 6.29 points improvement over DAPO.

Abstract

Reinforcement Learning from Verifiable Rewards (RLVR) typically samples multiple responses per prompt and assigns binary rewards based on individual correctness, yet the collective structure of the group output, specifically the distribution of errors, is largely discarded. We identify this as a missed opportunity: empirical analysis reveals that error diversity within a group is a strong predictor of training success, with problems eliciting diverse wrong answers benefiting substantially more from RLVR than those producing homogeneous failures. Motivated by this observation, we propose Error Diversity Advantage Shaping (EDAS), a lightweight, algorithm-agnostic technique that modulates the advantage signal for incorrect rollouts based on intra-group error diversity. EDAS amplifies penalties for dominant, repeated errors and attenuates penalties for rare, exploratory ones, thereby encouraging the model to maintain diverse reasoning paths and discouraging error perseveration. Crucially, EDAS operates as a simple post-hoc adjustment that can be seamlessly integrated into any RLVR algorithm. We validate EDAS on top of several mainstream RLVR methods across a series of models and seven challenging math benchmarks, demonstrating consistent improvements. Notably, EDAS yields an average improvement of 6.29 points over DAPO on Qwen3-8B across seven benchmarks, confirming that exploiting the latent information in group rollouts is a broadly effective strategy for strengthening RLVR.