Chatting with Images for Introspective Visual Thinking

TL;DR

This paper introduces 'chatting with images', a novel framework for visual reasoning that uses language prompts to dynamically re-encode and manipulate multiple image regions, improving multi-image and video reasoning in vision-language models.

Contribution

The paper proposes a new paradigm and model, ViLaVT, that enhances visual reasoning by integrating language-guided feature modulation and dynamic encoding, trained with a curriculum of supervised and reinforcement learning.

Findings

ViLaVT outperforms existing models on eight benchmarks.

Significant improvements on multi-image and video spatial reasoning tasks.

Effective joint re-encoding enhances cross-modal alignment.

Abstract

Current large vision-language models (LVLMs) typically rely on text-only reasoning based on a single-pass visual encoding, which often leads to loss of fine-grained visual information. Recently the proposal of ''thinking with images'' attempts to alleviate this limitation by manipulating images via external tools or code; however, the resulting visual states are often insufficiently grounded in linguistic semantics, impairing effective cross-modal alignment - particularly when visual semantics or geometric relationships must be reasoned over across distant regions or multiple images. To address these challenges, we propose ''chatting with images'', a new framework that reframes visual manipulation as language-guided feature modulation. Under the guidance of expressive language prompts, the model dynamically performs joint re-encoding over multiple image regions, enabling tighter coupling between linguistic reasoning and visual state updates. We instantiate this paradigm in ViLaVT, a novel LVLM equipped with a dynamic vision encoder explicitly designed for such interactive visual reasoning, and trained it with a two-stage curriculum combining supervised fine-tuning and reinforcement learning to promote effective reasoning behaviors. Extensive experiments across eight benchmarks demonstrate that ViLaVT achieves strong and consistent improvements, with particularly pronounced gains on complex multi-image and video-based spatial reasoning tasks.