Visual Thoughts: A Unified Perspective of Understanding Multimodal Chain-of-Thought
Zihui Cheng, Qiguang Chen, Xiao Xu, Jiaqi Wang, Weiyun Wang, Hao Fei, Yidong Wang, Alex Jinpeng Wang, Zhi Chen, Wanxiang Che, Libo Qin

TL;DR
This paper investigates how visual thoughts enhance multimodal chain-of-thought reasoning in large vision-language models, revealing their role as intermediaries that improve interpretability and performance across different MCoT formats.
Contribution
It systematically analyzes the forms and functions of visual thoughts, clarifies their mechanism in boosting MCoT, and introduces a comprehensive framework for understanding visual thought expressions.
Findings
Visual thoughts improve MCoT performance regardless of format.
Four distinct visual thought expressions differ in clarity and effectiveness.
Visual thoughts act as intermediaries transmitting visual information to deeper model layers.
Abstract
Large Vision-Language Models (LVLMs) have achieved significant success in multimodal tasks, with multimodal chain-of-thought (MCoT) further enhancing performance and interpretability. Recent MCoT methods fall into two categories: (i) Textual-MCoT (T-MCoT), which takes multimodal input and produces textual output; and (ii) Interleaved-MCoT (I-MCoT), which generates interleaved image-text outputs. Despite advances in both approaches, the mechanisms driving these improvements are not fully understood. To fill this gap, we first reveal that MCoT boosts LVLMs by incorporating visual thoughts, which convey image information to the reasoning process regardless of the MCoT format, depending only on clarity and conciseness of expression. Furthermore, to explore visual thoughts systematically, we define four distinct forms of visual thought expressions and analyze them comprehensively. Our…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
Taxonomy
TopicsLanguage, Metaphor, and Cognition
