Reliable Thinking with Images

TL;DR

This paper addresses the challenge of noisy reasoning in multimodal large language models by proposing a method that estimates the reliability of visual and textual cues to improve reasoning accuracy.

Contribution

It introduces RTWI, a novel approach that filters and votes on visual and textual cues to mitigate noise in multimodal reasoning tasks.

Findings

RTWI outperforms existing methods on seven benchmarks.

Effective filtering reduces error propagation in multimodal reasoning.

Reliability estimation improves overall reasoning accuracy.

Abstract

As a multimodal extension of Chain-of-Thought (CoT), Thinking with Images (TWI) has recently emerged as a promising avenue to enhance the reasoning capability of Multi-modal Large Language Models (MLLMs), which generates interleaved CoT by incorporating visual cues into the textual reasoning process. However, the success of existing TWI methods heavily relies on the assumption that interleaved image-text CoTs are faultless, which is easily violated in real-world scenarios due to the complexity of multimodal understanding. In this paper, we reveal and study a highly-practical yet under-explored problem in TWI, termed Noisy Thinking (NT). Specifically, NT refers to the imperfect visual cues mining and answer reasoning process. As the saying goes, ``One mistake leads to another'', erroneous interleaved CoT would cause error accumulation, thus significantly degrading the performance of MLLMs. To solve the NT problem, we propose a novel method dubbed Reliable Thinking with Images (RTWI). In brief, RTWI estimates the reliability of visual cues and textual CoT in a unified text-centric manner and accordingly employs robust filtering and voting modules to prevent NT from contaminating the final answer. Extensive experiments on seven benchmarks verify the effectiveness of RTWI against NT.