Retrieval-Augmented Visual Question Answering via Built-in Autoregressive Search Engines

TL;DR

ReAuSE is a unified model that integrates retrieval and answer generation for knowledge-based visual question answering, improving accuracy by combining a built-in search engine with relevance feedback.

Contribution

The paper introduces ReAuSE, a novel integrated model that combines retrieval and generation in a single multimodal large language model for VQA.

Findings

Achieves 2.9% to 9.6% performance improvements on OKVQA and A-OKVQA datasets.

Effectively integrates retrieval and answer generation in a unified framework.

Utilizes relevance feedback to enhance retrieval accuracy.

Abstract

Retrieval-augmented generation (RAG) has emerged to address the knowledge-intensive visual question answering (VQA) task. Current methods mainly employ separate retrieval and generation modules to acquire external knowledge and generate answers, respectively. We propose ReAuSE, an alternative to the previous RAG model for the knowledge-based VQA task, which seamlessly integrates knowledge retriever into the generative multi-modal large language model, serving as a built-in search engine. Specifically, our model functions both as a generative retriever and an accurate answer generator. It not only helps retrieve documents from the knowledge base by producing identifiers for each document, but it also answers visual questions based on the retrieved documents. Furthermore, we propose a reinforced retrieval calibration module from relevance feedback to improve retrieval performance and align with the preferences for accurate answer generation. Extensive experiments on two representative OKVQA and A-OKVQA datasets demonstrate significant improvements ranging from 2.9\% to 9.6\% across all evaluation metrics when compared to strong baselines.