Mitigating Think-Answer Mismatch in LLM Reasoning Through Noise-Aware Advantage Reweighting

TL;DR

This paper introduces S-GRPO, a noise-aware enhancement to group-relative policy optimization, significantly improving the robustness and performance of large reasoning models under noisy reward signals.

Contribution

We propose S-GRPO, a novel method that derives optimal advantage weights to stabilize training in the presence of reward noise, outperforming existing approaches.

Findings

S-GRPO achieves +2.5% to +2.4% performance gains on various benchmarks.

S-GRPO maintains stable learning under 20% synthetic reward noise.

Standard GRPO fails with high noise, while S-GRPO remains effective.

Abstract

Group-Relative Policy Optimization (GRPO) is a key technique for training large reasoning models, yet it suffers from a critical vulnerability: the \emph{Think-Answer Mismatch}, where noisy reward signals corrupt the learning process. This problem is most severe in unbalanced response groups, paradoxically degrading the signal precisely when it should be most informative. To address this challenge, we propose Stable Group-Relative Policy Optimization (S-GRPO), a principled enhancement that derives optimal, noise-aware advantage weights to stabilize training. Our comprehensive experiments on mathematical reasoning benchmarks demonstrate S-GRPO's effectiveness and robustness. On various models, S-GRPO significantly outperforms DR. GRPO, achieving performance gains of +2.5% on Qwen-Math-7B-Base, +2.2% on Llama-3.2-3B-Base, and +2.4% on Qwen-Math-1.5B-Instruct. Most critically, while standard GRPO fails to learn under 20% synthetic reward noise, S-GRPO maintains stable learning progress. These results highlight S-GRPO's potential for more robust and effective training of large-scale reasoning models. \footnote{Code and data are available at: https://github.com/shenpeijun0212/S-GRPO