Enhancing Visual Question Answering with Multimodal LLMs via Chain-of-Question Guided Retrieval-Augmented Generation

TL;DR

This paper introduces a novel retrieval-augmented framework for multimodal large language models to improve visual question answering by integrating structured reasoning and external knowledge.

Contribution

It proposes a new logical prompting strategy combining Chain-of-Thought and Visual Question Decomposition to guide knowledge retrieval in VQA tasks.

Findings

Enhanced accuracy on E-VQA, InfoSeek, and OKVQA benchmarks.

Improved reasoning coherence and knowledge relevance in VQA.

Better generalization in complex cross-domain scenarios.

Abstract

With advances in multimodal research and deep learning, Multimodal Large Language Models (MLLMs) have emerged as a powerful paradigm for a wide range of multimodal tasks. As a core problem in vision-language research, Visual Question Answering (VQA) has increasingly employed MLLMs to improve performance, particularly in open-domain settings where external knowledge is essential. In this work, we aim to further enhance retrieval-based VQA by more effectively integrating MLLMs with structured reasoning and knowledge acquisition. We introduce a logical prompting strategy that fuses Chain-of-Thought (CoT) reasoning with Visual Question Decomposition (VQD), termed CoVQD, to guide retrieval toward more accurate and relevant knowledge for MLLM inference. Building on this idea, we propose a new framework, CoVQD-guided RAG (CgRAG), which enables MLLMs to access more comprehensive and coherent external knowledge while benefiting from structured visual-text reasoning guidance, thereby improving generalization and reliability in complex cross-domain VQA scenarios. Extensive experiments on E-VQA, InfoSeek, and OKVQA benchmarks demonstrate the effectiveness of the proposed method.