Socratic Questioning: Learn to Self-guide Multimodal Reasoning in the Wild

TL;DR

This paper introduces Socratic Questioning, a novel multi-round training framework for lightweight multimodal models that improves visual reasoning, reduces hallucinations, and enhances fine-grained image understanding through self-guided questioning.

Contribution

The paper proposes a new self-questioning framework called Socratic Questioning for multimodal models, combining reasoning and instruction tuning to improve visual reasoning and hallucination mitigation.

Findings

31.2% reduction in hallucination score

Significant improvement in zero-shot visual reasoning

Effective in complex visual question-answering tasks

Abstract

Complex visual reasoning remains a key challenge today. Typically, the challenge is tackled using methodologies such as Chain of Thought (COT) and visual instruction tuning. However, how to organically combine these two methodologies for greater success remains unexplored. Also, issues like hallucinations and high training cost still need to be addressed. In this work, we devise an innovative multi-round training and reasoning framework suitable for lightweight Multimodal Large Language Models (MLLMs). Our self-questioning approach heuristically guides MLLMs to focus on visual clues relevant to the target problem, reducing hallucinations and enhancing the model's ability to describe fine-grained image details. This ultimately enables the model to perform well in complex visual reasoning and question-answering tasks. We have named this framework Socratic Questioning(SQ). To facilitate future research, we create a multimodal mini-dataset named CapQA, which includes 1k images of fine-grained activities, for visual instruction tuning and evaluation, our proposed SQ method leads to a 31.2% improvement in the hallucination score. Our extensive experiments on various benchmarks demonstrate SQ's remarkable capabilities in heuristic self-questioning, zero-shot visual reasoning and hallucination mitigation. Our model and code will be publicly available.