STAPO: Stabilizing Reinforcement Learning for LLMs by Silencing Rare Spurious Tokens

TL;DR

This paper introduces STAPO, a method to stabilize reinforcement learning fine-tuning of large language models by silencing rare spurious tokens that cause instability and performance collapse.

Contribution

The paper proposes a unified framework to analyze token-level impacts and introduces S2T and STAPO to suppress spurious tokens, improving training stability and performance.

Findings

STAPO achieves an average performance improvement of 11.49% over baseline methods.

STAPO enhances entropy stability during RL fine-tuning.

Spurious tokens are identified as a key factor in training instability.

Abstract

Reinforcement Learning (RL) has significantly improved large language model reasoning, but existing RL fine-tuning methods rely heavily on heuristic techniques such as entropy regularization and reweighting to maintain stability. In practice, they often suffer from late-stage performance collapse, leading to degraded reasoning quality and unstable training. We identify a key factor behind this instability: a small fraction of tokens, termed spurious tokens (around 0.01%), which contribute little to the reasoning outcome but receive disproportionately amplified gradient updates due to inheriting the full sequence-level reward. We present a unified framework for evaluating token-level optimization impacts across spurious risk, gradient norms, and entropy changes. Building on the analysis of token characteristics that severely disrupt optimization, we propose the Silencing Spurious Tokens (S2T) mechanism to efficiently suppress their gradient perturbations. Incorporating this mechanism into a group-based objective, we propose Spurious-Token-Aware Policy Optimization (STAPO), which promotes stable and effective large-scale model refinement. Across six mathematical reasoning benchmarks using Qwen 1.7B, 8B, and 14B base models, STAPO consistently demonstrates superior entropy stability and achieves an average performance improvement of 11.49% ($\rho_{\mathrm{T}}$=1.0, top-p=1.0) and 3.73% ($\rho_{\mathrm{T}}$=0.7, top-p=0.9) over GRPO, 20-Entropy, and JustRL.