ProReason: Multi-Modal Proactive Reasoning with Decoupled Eyesight and Wisdom

TL;DR

ProReason introduces a novel multi-modal visual reasoning framework that separates visual perception from textual reasoning, enabling iterative proactive perception and improved performance on visual reasoning benchmarks.

Contribution

It proposes a decoupled vision-reasoning framework with multi-run proactive perception, enhancing multi-modal reasoning and integrating large language models effectively.

Findings

Outperforms existing multi-step reasoning frameworks with an average gain of 13.2%.

Enables high-quality visual reasoning data generation for downstream tasks.

Achieves superior performance on various benchmarks for open-source and closed-source models.

Abstract

Large vision-language models (LVLMs) have witnessed significant progress on visual understanding tasks. However, they often prioritize language knowledge over image information on visual reasoning tasks, incurring performance degradation. To tackle this issue, we first identify the drawbacks of existing solutions (i.e., limited multi-modal reasoning capacities, and insufficient and irrelevant visual descriptions). We then decompose visual reasoning process into two stages: proactive visual perception (i.e., eyesight) and textual reasoning (i.e., wisdom), and introduce a novel visual reasoning framework named ProReason. This framework features decoupled vision-reasoning capabilities and multi-run proactive perception. Briefly, given a multi-modal question, ProReason iterates proactive information collection and reasoning until the answer can be concluded with necessary and sufficient visual descriptions. Notably, the disassociation of capabilities allows seamless integration of existing large language models (LLMs) to compensate for the reasoning deficits of LVLMs. Our extensive experiments demonstrate that ProReason outperforms existing multi-step reasoning frameworks on various benchmarks for both open-source and closed-source models, with the average performance gain reaching 13.2%. Besides, the integration of LLMs allows ProReason to produce high-quality visual reasoning data, which empowers ProReason-distilled models (i.e., ProReason-VL and ProReason-Q3) to achieve superior performance in downstream tasks. Our insights into existing solutions and the decoupled perspective for feasible integration of LLMs illuminate future research on visual reasoning techniques, especially LLM-assisted ones.