Thinking with Deltas: Incentivizing Reinforcement Learning via Differential Visual Reasoning Policy

TL;DR

This paper introduces a novel reinforcement learning framework called Thinking with Deltas, which incentivizes models to focus on visual evidence by maximizing reasoning divergence from masked inputs and minimizing divergence from perturbed inputs, thereby improving visual reasoning.

Contribution

The paper proposes the Differential Visual Reasoning Policy (DVRP), a new intrinsic supervision method that enhances visual understanding in multimodal models by aligning reasoning with visual information changes.

Findings

DVRP significantly outperforms state-of-the-art methods on various benchmarks.

Models trained with DVRP demonstrate increased visual sensitivity and robustness.

The approach improves reasoning in both general and medical domains.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced reasoning capabilities in Large Language Models. However, adapting RLVR to multimodal domains suffers from a critical \textit{perception-reasoning decoupling}. Existing paradigms, driven by text-centric outcome rewards, reasoning in language medium, inadvertently encourage models to bypass visual perception. We empirically validate this through blind experiments: state-of-the-art policies maintain or surprisingly improve performance even when visual inputs are entirely removed. This reveals that these models degenerate into \textit{blind reasoners}, exploiting linguistic priors to generate plausible answers instead of attending to visual evidence. In response, we propose \textbf{Thinking with Deltas}, a framework driven by a \textbf{Differential Visual Reasoning Policy (DVRP)}. DVRP introduces intrinsic supervision via visual triplets, comprising original, masked, and perturbed inputs. It optimizes the model to maximize reasoning divergence from masked inputs (enforcing \textit{visual sensitivity}) while minimizing divergence from perturbed inputs (ensuring \textit{visual robustness}). By aligning reasoning variations strictly with the \textit{Delta} of visual information, DVRP inherently bolsters visual understanding capabilities and significantly outperforms state-of-the-art methods on both general and medical benchmarks, without requiring external annotations or auxiliary tools.