Co-VQA : Answering by Interactive Sub Question Sequence

TL;DR

This paper introduces Co-VQA, a conversation-based framework for visual question answering that decomposes complex questions into sub questions, enabling more interpretable and accurate reasoning, achieving state-of-the-art results on VQA-CP v2.

Contribution

It proposes a novel interactive VQA framework with sub question sequences, an adaptive reasoning model, and a new dataset construction method for supervised learning.

Findings

Achieves state-of-the-art on VQA-CP v2.

Sub question sequences improve interpretability and reasoning.

Explicit question-image semantic connections are built.

Abstract

Most existing approaches to Visual Question Answering (VQA) answer questions directly, however, people usually decompose a complex question into a sequence of simple sub questions and finally obtain the answer to the original question after answering the sub question sequence(SQS). By simulating the process, this paper proposes a conversation-based VQA (Co-VQA) framework, which consists of three components: Questioner, Oracle, and Answerer. Questioner raises the sub questions using an extending HRED model, and Oracle answers them one-by-one. An Adaptive Chain Visual Reasoning Model (ACVRM) for Answerer is also proposed, where the question-answer pair is used to update the visual representation sequentially. To perform supervised learning for each model, we introduce a well-designed method to build a SQS for each question on VQA 2.0 and VQA-CP v2 datasets. Experimental results show that our method achieves state-of-the-art on VQA-CP v2. Further analyses show that SQSs help build direct semantic connections between questions and images, provide question-adaptive variable-length reasoning chains, and with explicit interpretability as well as error traceability.