WikiSeeker: Rethinking the Role of Vision-Language Models in Knowledge-Based Visual Question Answering

TL;DR

WikiSeeker introduces a novel multi-modal RAG framework for KB-VQA that leverages Vision-Language Models more effectively through specialized agents, achieving state-of-the-art results.

Contribution

It redefines the role of VLMs in KB-VQA by introducing a multi-modal retriever and specialized agents, enhancing retrieval and answer accuracy.

Findings

Achieves state-of-the-art performance on EVQA, InfoSeek, and M2KR datasets.

Significantly improves retrieval accuracy and answer quality.

Utilizes VLMs as Refiner and Inspector for better context handling.

Abstract

Multi-modal Retrieval-Augmented Generation (RAG) has emerged as a highly effective paradigm for Knowledge-Based Visual Question Answering (KB-VQA). Despite recent advancements, prevailing methods still primarily depend on images as the retrieval key, and often overlook or misplace the role of Vision-Language Models (VLMs), thereby failing to leverage their potential fully. In this paper, we introduce WikiSeeker, a novel multi-modal RAG framework that bridges these gaps by proposing a multi-modal retriever and redefining the role of VLMs. Rather than serving merely as answer generators, we assign VLMs two specialized agents: a Refiner and an Inspector. The Refiner utilizes the capability of VLMs to rewrite the textual query according to the input image, significantly improving the performance of the multimodal retriever. The Inspector facilitates a decoupled generation strategy by selectively routing reliable retrieved context to another LLM for answer generation, while relying on the VLM's internal knowledge when retrieval is unreliable. Extensive experiments on EVQA, InfoSeek, and M2KR demonstrate that WikiSeeker achieves state-of-the-art performance, with substantial improvements in both retrieval accuracy and answer quality. Our code will be released on https://github.com/zhuyjan/WikiSeeker.