Knowledge Condensation and Reasoning for Knowledge-based VQA

TL;DR

This paper introduces a novel approach for knowledge-based visual question answering that condenses external knowledge through multimodal perception and large language models, leading to state-of-the-art results without relying on GPT-3 generated knowledge.

Contribution

The paper proposes two synergistic models for knowledge condensation and reasoning, improving relevance and accuracy in KB-VQA tasks over previous methods.

Findings

Achieves 65.1% on OK-VQA and 60.1% on A-OKVQA datasets

Outperforms previous methods without GPT-3 knowledge

Demonstrates effective knowledge condensation and reasoning

Abstract

Knowledge-based visual question answering (KB-VQA) is a challenging task, which requires the model to leverage external knowledge for comprehending and answering questions grounded in visual content. Recent studies retrieve the knowledge passages from external knowledge bases and then use them to answer questions. However, these retrieved knowledge passages often contain irrelevant or noisy information, which limits the performance of the model. To address the challenge, we propose two synergistic models: Knowledge Condensation model and Knowledge Reasoning model. We condense the retrieved knowledge passages from two perspectives. First, we leverage the multimodal perception and reasoning ability of the visual-language models to distill concise knowledge concepts from retrieved lengthy passages, ensuring relevance to both the visual content and the question. Second, we leverage the text comprehension ability of the large language models to summarize and condense the passages into the knowledge essence which helps answer the question. These two types of condensed knowledge are then seamlessly integrated into our Knowledge Reasoning model, which judiciously navigates through the amalgamated information to arrive at the conclusive answer. Extensive experiments validate the superiority of the proposed method. Compared to previous methods, our method achieves state-of-the-art performance on knowledge-based VQA datasets (65.1% on OK-VQA and 60.1% on A-OKVQA) without resorting to the knowledge produced by GPT-3 (175B).