Scaling Agentic Reinforcement Learning for Tool-Integrated Reasoning in VLMs

TL;DR

This paper introduces VISTA-Gym, a scalable training environment that enhances vision-language models' ability to reason through multi-step visual interactions by integrating tools and reinforcement learning, leading to significant performance improvements.

Contribution

The paper presents VISTA-Gym, a unified platform for training VLMs with tool-integrated reasoning using reinforcement learning, enabling models to better handle complex visual reasoning tasks.

Findings

VISTA-R1-8B outperforms state-of-the-art baselines by 9.51%-18.72%.

VISTA-Gym effectively trains models for multi-step visual reasoning.

Models trained with VISTA-Gym show improved tool use and reasoning capabilities.

Abstract

While recent vision-language models (VLMs) demonstrate strong image understanding, their ability to "think with images", i.e., to reason through multi-step visual interactions, remains limited. We introduce VISTA-Gym, a scalable training environment for incentivizing tool-integrated visual reasoning capabilities in VLMs. VISTA-Gym unifies diverse real-world multimodal reasoning tasks (7 tasks from 13 datasets in total) with a standardized interface for visual tools (e.g., grounding, parsing), executable interaction loops, verifiable feedback signals, and efficient trajectory logging, enabling visual agentic reinforcement learning at scale. While recent VLMs exhibit strong text-only reasoning, both proprietary and open-source models still struggle with tool selection, invocation, and coordination. With VISTA-Gym, we train VISTA-R1 to interleave tool-use with agentic reasoning via multi-turn trajectory sampling and end-to-end reinforcement learning. Extensive experiments across 11 public reasoning-intensive VQA benchmarks show that VISTA-R1-8B outperforms state-of-the-art baselines with similar sizes by 9.51%-18.72%, demonstrating VISTA-Gym as an effective training ground to unlock the tool-integrated reasoning capabilities for VLMs.