Advancing Multimodal Reasoning: From Optimized Cold Start to Staged Reinforcement Learning

TL;DR

This paper introduces ReVisual-R1, a staged training method for multimodal large language models that significantly improves reasoning capabilities by optimizing cold start initialization and addressing reinforcement learning challenges.

Contribution

The paper presents a novel staged training pipeline for MLLMs, emphasizing cold start initialization and overcoming gradient stagnation in multimodal RL, leading to state-of-the-art results.

Findings

Careful text data initialization boosts reasoning performance.

Gradient stagnation hampers multimodal RL training stability.

Sequential text-only RL further enhances multimodal reasoning.

Abstract

Inspired by the remarkable reasoning capabilities of Deepseek-R1 in complex textual tasks, many works attempt to incentivize similar capabilities in Multimodal Large Language Models (MLLMs) by directly applying reinforcement learning (RL). However, they still struggle to activate complex reasoning. In this paper, rather than examining multimodal RL in isolation, we delve into current training pipelines and identify three crucial phenomena: 1) Effective cold start initialization is critical for enhancing MLLM reasoning. Intriguingly, we find that initializing with carefully selected text data alone can lead to performance surpassing many recent multimodal reasoning models, even before multimodal RL. 2) Standard GRPO applied to multimodal RL suffers from gradient stagnation, which degrades training stability and performance. 3) Subsequent text-only RL training, following the multimodal RL phase, further enhances multimodal reasoning. This staged training approach effectively balances perceptual grounding and cognitive reasoning development. By incorporating the above insights and addressing multimodal RL issues, we introduce ReVisual-R1, achieving a new state-of-the-art among open-source 7B MLLMs on challenging benchmarks including MathVerse, MathVision, WeMath, LogicVista, DynaMath, and challenging AIME2024 and AIME2025.