Deep But Reliable: Advancing Multi-turn Reasoning for Thinking with Images

TL;DR

This paper introduces DRIM, a multimodal reasoning model that enhances multi-turn visual reasoning by incorporating self-reflection and correction mechanisms, leading to improved performance on visual understanding tasks.

Contribution

The paper presents a novel multi-stage training pipeline for deep, reliable reasoning with images, including data construction, supervised fine-tuning, and reinforcement learning with redundancy penalties.

Findings

DRIM outperforms existing models on visual reasoning benchmarks.

The model effectively self-reflects and corrects reasoning errors.

Redundancy-penalized policy optimization improves reasoning reliability.

Abstract

Recent advances in large Vision-Language Models (VLMs) have exhibited strong reasoning capabilities on complex visual tasks by thinking with images in their Chain-of-Thought (CoT), which is achieved by actively invoking tools to analyze visual inputs rather than merely perceiving them. However, existing models often struggle to reflect on and correct themselves when attempting incorrect reasoning trajectories. To address this limitation, we propose DRIM, a model that enables deep but reliable multi-turn reasoning when thinking with images in its multimodal CoT. Our pipeline comprises three stages: data construction, cold-start SFT and RL. Based on a high-resolution image dataset, we construct high-difficulty and verifiable visual question-answer pairs, where solving each task requires multi-turn tool calls to reach the correct answer. In the SFT stage, we collect tool trajectories as cold-start data, guiding a multi-turn reasoning pattern. In the RL stage, we introduce redundancy-penalized policy optimization, which incentivizes the model to develop a self-reflective reasoning pattern. The basic idea is to impose judgment on reasoning trajectories and penalize those that produce incorrect answers without sufficient multi-scale exploration. Extensive experiments demonstrate that DRIM achieves superior performance on visual understanding benchmarks.