Simple o3: Towards Interleaved Vision-Language Reasoning

TL;DR

Simple o3 introduces an end-to-end multimodal reasoning framework that integrates dynamic visual operations with linguistic reasoning, significantly improving vision-language task performance through a scalable data synthesis pipeline and interleaved reasoning strategies.

Contribution

The paper presents Simple o3, a novel approach combining visual transformations and linguistic reasoning with a new data synthesis pipeline and analysis of interleaved reasoning strategies.

Findings

Enhanced reasoning with additional visual tokens improves performance.

Reusing and magnifying images boosts visual reasoning and perception.

Cropping images based on visual grounding enhances focus on key regions.

Abstract

Multimodal Large Language Models (MLLMs) have shown impressive performance on vision-language tasks, but their long Chain-of-Thought (CoT) capabilities in multimodal scenarios remain underexplored. Inspired by OpenAI's o3 model, which emulates human-like ''thinking with image'' through iterative visual transformations and linguistic reasoning, we propose Simple o3, an end-to-end framework that integrates dynamic tool interactions (e.g., cropping, zooming, and reusing) into interleaved vision-language reasoning via supervised fine-tuning (SFT). Our approach features a scalable data synthesis pipeline that generates high-quality interleaved vision-language reasoning chains via an ''observe-reason-act'' cycle, complete with executable visual operations and rigorous verification, yielding the open-source TWI-Tools-146K dataset. Experimental results demonstrate Simple o3's superior performance on diverse benchmarks, outperforming existing approaches. By combining enhanced reasoning capabilities, Simple o3 establishes a powerful yet computationally affordable paradigm for advancing multimodal reasoning. Remarkably, we provide the first in-depth analysis of different interleaved reasoning strategies, offering insights into their impact on model performance. We found that by introducing additional visual tokens for interleaved vision-language reasoning, reusing and magnifying the original image significantly improves the model's visual reasoning and fine-grained perception, while image cropping based on precise visual grounding allows the model to effectively focus on key entities or regions, further enhancing its capabilities.