Stable and Efficient Single-Rollout RL for Multimodal Reasoning

TL;DR

This paper introduces MSSR, a novel single-rollout reinforcement learning framework for multimodal reasoning that enhances training stability and efficiency, outperforming existing group-based methods across multiple benchmarks.

Contribution

MSSR is a group-free RLVR method with an entropy-based advantage-shaping mechanism that stabilizes training and improves performance in multimodal reasoning tasks.

Findings

MSSR achieves similar accuracy with half the training steps of group-based methods.

MSSR surpasses group-based baselines when trained for the same number of steps.

MSSR shows consistent generalization across five reasoning benchmarks.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has become a key paradigm to improve the reasoning capabilities of Multimodal Large Language Models (MLLMs). However, prevalent group-based algorithms such as GRPO require multi-rollout sampling for each prompt. While more efficient single-rollout variants have recently been explored in text-only settings, we find that they suffer from severe instability in multimodal contexts, often leading to training collapse. To address this training efficiency-stability trade-off, we introduce $\textbf{MSSR}$ (Multimodal Stabilized Single-Rollout), a group-free RLVR framework that achieves both stable optimization and effective multimodal reasoning performance. MSSR achieves this via an entropy-based advantage-shaping mechanism that adaptively regularizes advantage magnitudes, preventing collapse and maintaining training stability. While such mechanisms have been used in group-based RLVR, we show that in the multimodal single-rollout setting they are not merely beneficial but essential for stability. In in-distribution evaluations, MSSR demonstrates superior training compute efficiency, achieving similar validation accuracy to the group-based baseline with half the training steps. When trained for the same number of steps, MSSR's performance surpasses the group-based baseline and shows consistent generalization improvements across five diverse reasoning-intensive benchmarks. Together, these results demonstrate that MSSR enables stable, compute-efficient, and effective RLVR for complex multimodal reasoning tasks.