Multiple interaction learning with question-type prior knowledge for constraining answer search space in visual question answering

TL;DR

This paper introduces a novel VQA model that leverages question-type prior knowledge and multiple modality interactions to better constrain answer search space, improving accuracy on benchmark datasets.

Contribution

It proposes a new VQA approach that utilizes question-type priors and multiple modality interactions, which was not explored in prior works.

Findings

Achieves state-of-the-art performance on VQA 2.0 and TDIUC datasets.

Demonstrates the effectiveness of question-type priors in constraining answer search.

Shows that multiple modality interactions improve VQA accuracy.

Abstract

Different approaches have been proposed to Visual Question Answering (VQA). However, few works are aware of the behaviors of varying joint modality methods over question type prior knowledge extracted from data in constraining answer search space, of which information gives a reliable cue to reason about answers for questions asked in input images. In this paper, we propose a novel VQA model that utilizes the question-type prior information to improve VQA by leveraging the multiple interactions between different joint modality methods based on their behaviors in answering questions from different types. The solid experiments on two benchmark datasets, i.e., VQA 2.0 and TDIUC, indicate that the proposed method yields the best performance with the most competitive approaches.