SynthRL: Scaling Visual Reasoning with Verifiable Data Synthesis

TL;DR

SynthRL is a scalable data synthesis pipeline that enhances visual reasoning models by generating verifiable, challenging questions, leading to improved performance on out-of-domain benchmarks.

Contribution

We introduce SynthRL, a novel pipeline for automatic, verifiable data augmentation in reasoning-oriented RL training, improving model generalization and reasoning complexity.

Findings

SynthRL generated over 3.3K verifiable questions from 8K seed samples.

Models trained with SynthRL data outperformed baselines on five reasoning benchmarks.

Performance gains were especially notable on the most challenging samples.

Abstract

Vision-language models (VLMs) trained via reinforcement learning with verifiable reward (RLVR) have shown notable progress in scaling test-time compute effectively. In this work, we investigate how synthesized RL data can further improve RLVR. To this end, we propose \textbf{SynthRL}-a scalable and guaranteed pipeline for automatic data scaling in reasoning-oriented RL training. SynthRL comprises three key stages: (1) selecting seed questions with appropriate distribution, (2) augmenting them into more challenging variants while preserving the original answers, and (3) a guaranteed verification stage that ensures near-perfect correctness and difficulty enhancement. Our empirical experiments demonstrate SynthRL's scalability and effectiveness. When applied to the MMK12 dataset, SynthRL synthesizes over 3.3K additional verifiable, challenging questions from approximately 8K seed samples. Models trained with our synthesized data achieve consistent gains across five out-of-domain visual math reasoning benchmarks, with a significant improvement over baseline models trained on seed data alone. Notably, detailed analysis reveals that the gains are more pronounced on the most challenging evaluation samples, highlighting SynthRL's effectiveness in eliciting deeper and more complex reasoning patterns.