Shuffle-R1: Efficient RL framework for Multimodal Large Language Models via Data-centric Dynamic Shuffle

TL;DR

Shuffle-R1 is a novel RL framework that enhances training efficiency for multimodal large language models by dynamically restructuring data sampling and batch composition, addressing key issues like advantage collapsing and rollout silencing.

Contribution

It introduces data-centric techniques including pairwise trajectory sampling and advantage-based shuffling to improve gradient signals and training efficiency in RL fine-tuning of MLLMs.

Findings

Outperforms strong RL baselines on reasoning benchmarks

Achieves higher gradient quality with minimal overhead

Demonstrates improved long-term learning efficiency

Abstract

Reinforcement learning (RL) has emerged as an effective post-training paradigm for enhancing the reasoning capabilities of multimodal large language model (MLLM). However, current RL pipelines often suffer from training inefficiencies caused by two underexplored issues: Advantage Collapsing, where most advantages in a batch concentrate near zero, and Rollout Silencing, where the proportion of rollouts contributing non-zero gradients diminishes over time. These issues lead to suboptimal gradient updates and hinder long-term learning efficiency. To address these issues, we propose Shuffle-R1, a simple yet principled framework that improves RL fine-tuning efficiency by dynamically restructuring trajectory sampling and batch composition. It introduces (1) Pairwise Trajectory Sampling, which selects high-contrast trajectories with large advantages to improve gradient signal quality, and (2) Advantage-based Trajectory Shuffle, which increases exposure of valuable rollouts through informed batch reshuffling. Experiments across multiple reasoning benchmarks show that our framework consistently outperforms strong RL baselines with minimal overhead. These results highlight the importance of data-centric adaptations for more efficient RL training in MLLM.