What Does Vision Tool-Use Reinforcement Learning Really Learn? Disentangling Tool-Induced and Intrinsic Effects for Crop-and-Zoom

TL;DR

This paper introduces MED, a framework to disentangle intrinsic learning from tool-induced effects in vision RL models, revealing that current methods mainly reduce harm rather than mastering tools.

Contribution

The paper presents MED, a novel analysis framework that separates intrinsic capabilities from tool effects, providing insights into how vision RL models learn to use tools.

Findings

Improvements are mainly due to intrinsic learning, not tool mastery.

Tool-use RL reduces tool-induced harm, such as errors and schema interference.

Current models coexist with tools but do not fully master them.

Abstract

Vision tool-use reinforcement learning (RL) can equip vision language models with visual operators such as crop-and-zoom and achieves strong performance gains, yet it remains unclear whether these gains are driven by improvements in tool use or evolving intrinsic capabilities. We introduce MED (Measure--Explain--Diagnose), a coarse-to-fine framework that disentangles intrinsic capability changes from tool-induced effects, decomposes the tool-induced performance difference into gain and harm terms, and probes the mechanisms driving their evolution. Across checkpoint-level analyses in the crop-and-zoom setting on two VLMs with different tool priors and six benchmarks, we find that improvements are dominated by intrinsic learning, while tool-use RL mainly reduces tool-induced harm (e.g., fewer call-induced errors and weaker tool schema interference) and yields limited progress in tool-based correction of intrinsic failures. Overall, in the crop-and-zoom setting studied here, current vision tool-use RL learns to coexist safely with tools rather than master them.