Interpretable Visual Question Answering Referring to Outside Knowledge

TL;DR

This paper introduces a multimodal interpretable VQA model that integrates outside knowledge and multiple captions to enhance answer accuracy and explanation diversity, outperforming existing methods.

Contribution

It presents a novel approach combining outside knowledge and multiple image captions for more rational and interpretable visual question answering.

Findings

Outperforms state-of-the-art in answer accuracy

Generates more diverse and rational explanations

Utilizes outside knowledge for improved reasoning

Abstract

We present a novel multimodal interpretable VQA model that can answer the question more accurately and generate diverse explanations. Although researchers have proposed several methods that can generate human-readable and fine-grained natural language sentences to explain a model's decision, these methods have focused solely on the information in the image. Ideally, the model should refer to various information inside and outside the image to correctly generate explanations, just as we use background knowledge daily. The proposed method incorporates information from outside knowledge and multiple image captions to increase the diversity of information available to the model. The contribution of this paper is to construct an interpretable visual question answering model using multimodal inputs to improve the rationality of generated results. Experimental results show that our model can outperform state-of-the-art methods regarding answer accuracy and explanation rationality.