What does RL improve for Visual Reasoning? A Frankenstein-Style Analysis

TL;DR

This paper investigates how reinforcement learning (RL) enhances visual reasoning in vision-language models, revealing that RL primarily refines mid-to-late transformer layers rather than improving core visual perception.

Contribution

The study introduces a Frankenstein-style analysis framework to dissect RL's effects, clarifying that RL's benefits stem from systematic mid-to-late layer refinements rather than general perception improvements.

Findings

RL causes a consistent inference-time shift in mid-to-late layers.

Mid-to-late layer refinements are transferable via merging.

RL improvements are necessary and not just coincidental.

Abstract

Reinforcement learning (RL) with verifiable rewards has become a standard post-training stage for boosting visual reasoning in vision-language models, yet it remains unclear what capabilities RL actually improves compared with supervised fine-tuning as cold-start initialization (IN). End-to-end benchmark gains conflate multiple factors, making it difficult to attribute improvements to specific skills. To bridge the gap, we propose a Frankenstein-style analysis framework including: (i) functional localization via causal probing; (ii) update characterization via parameter comparison; and (iii) transferability test via model merging. Instead, RL induces a consistent inference-time shift primarily in mid-to-late layers, and these mid-to-late refinements are both transferable (via merging) and necessary (via freezing) for RL gains. Overall, our results suggest that RL's reliable contribution in visual reasoning is not a uniform enhancement of visual perception, but a systematic refinement of mid-to-late transformer computation that improves vision-to-reasoning alignment and reasoning performance, highlighting the limitations of benchmark-only evaluation for understanding multimodal reasoning improvements.