Self-Bootstrapped Visual-Language Model for Knowledge Selection and Question Answering

TL;DR

This paper introduces a self-bootstrapped visual-language framework that improves knowledge selection and question answering in open-domain VQA by iteratively refining key knowledge retrieval and answer prediction, achieving state-of-the-art results.

Contribution

The proposed method leverages a visual-language model to select relevant knowledge and iteratively fine-tune both knowledge selection and answering modules, enhancing open-domain VQA performance.

Findings

Achieved 62.83% accuracy on OK-VQA benchmark.

Significantly outperforms baseline models.

Demonstrates effectiveness of self-bootstrapping in knowledge selection.

Abstract

While large visual-language models (LVLM) have shown promising results on traditional visual question answering benchmarks, it is still challenging for them to answer complex VQA problems which requires diverse world knowledge. Motivated by the research of retrieval-augmented generation in the field of natural language processing, we use Dense Passage Retrieval (DPR) to retrieve related knowledge to help the model answer questions. However, DPR conduct retrieving in natural language space, which may not ensure comprehensive acquisition of image information. Thus, the retrieved knowledge is not truly conducive to helping answer the question, affecting the performance of the overall system. To address this issue, we propose a novel framework that leverages the visual-language model to select the key knowledge retrieved by DPR and answer questions. The framework consists of two modules: Selector and Answerer, where both are initialized by the LVLM and parameter-efficiently finetuned by self-bootstrapping: find key knowledge in the retrieved knowledge documents using the Selector, and then use them to finetune the Answerer to predict answers; obtain the pseudo-labels of key knowledge documents based on the predictions of the Answerer and weak supervision labels, and then finetune the Selector to select key knowledge; repeat. Our framework significantly enhances the performance of the baseline on the challenging open-domain Knowledge-based VQA benchmark, OK-VQA, achieving a state-of-the-art accuracy of 62.83%. Our code is publicly available at https://github.com/haodongze/Self-KSel-QAns.