Multimodal Reranking for Knowledge-Intensive Visual Question Answering

TL;DR

This paper introduces a multi-modal reranker to improve the relevance scoring of knowledge candidates in visual question answering, leading to better answer generation by leveraging cross-modal interactions.

Contribution

It proposes a novel multi-modal reranking module that enhances candidate relevance scoring using cross-item interactions, addressing limitations of previous retrieval methods.

Findings

Improves answer accuracy on OK-VQA and A-OKVQA datasets.

Multi-modal reranker yields consistent performance gains.

Training with noisier candidates enhances test performance.

Abstract

Knowledge-intensive visual question answering requires models to effectively use external knowledge to help answer visual questions. A typical pipeline includes a knowledge retriever and an answer generator. However, a retriever that utilizes local information, such as an image patch, may not provide reliable question-candidate relevance scores. Besides, the two-tower architecture also limits the relevance score modeling of a retriever to select top candidates for answer generator reasoning. In this paper, we introduce an additional module, a multi-modal reranker, to improve the ranking quality of knowledge candidates for answer generation. Our reranking module takes multi-modal information from both candidates and questions and performs cross-item interaction for better relevance score modeling. Experiments on OK-VQA and A-OKVQA show that multi-modal reranker from distant supervision provides consistent improvements. We also find a training-testing discrepancy with reranking in answer generation, where performance improves if training knowledge candidates are similar to or noisier than those used in testing.