Thinking Before Looking: Improving Multimodal LLM Reasoning via Mitigating Visual Hallucination

TL;DR

This paper introduces the Visual Inference Chain (VIC), a novel framework that constructs reasoning chains using only textual context before visual input, significantly reducing hallucinations and improving reasoning accuracy in multimodal large language models.

Contribution

The paper proposes VIC, a new approach that mitigates visual hallucinations in MLLMs by separating textual reasoning from visual input, enhancing multimodal reasoning performance.

Findings

VIC reduces hallucinations caused by misleading images.

VIC improves zero-shot performance on vision-related tasks.

VIC enhances the reasoning capabilities of MLLMs.

Abstract

Multimodal large language models (MLLMs) have advanced the integration of visual and linguistic modalities, establishing themselves as the dominant paradigm for visual-language tasks. Current approaches like chain of thought (CoT) reasoning have augmented the cognitive capabilities of large language models (LLMs), yet their adaptation to MLLMs is hindered by heightened risks of hallucination in cross-modality comprehension. In this paper, we find that the thinking while looking paradigm in current multimodal CoT approaches--where reasoning chains are generated alongside visual input--fails to mitigate hallucinations caused by misleading images. To address these limitations, we propose the Visual Inference Chain (VIC) framework, a novel approach that constructs reasoning chains using textual context alone before introducing visual input, effectively reducing cross-modal biases and enhancing multimodal reasoning accuracy. Comprehensive evaluations demonstrate that VIC significantly improves zero-shot performance across various vision-related tasks, mitigating hallucinations while refining the reasoning capabilities of MLLMs. Our code repository can be found at https://github.com/Terry-Xu-666/visual_inference_chain.