iVQA: Inverse Visual Question Answering

TL;DR

This paper introduces iVQA, an inverse visual question answering task that requires generating questions from images and answers, serving as a new benchmark for visuo-linguistic understanding and proposing a novel model with a dynamic attention mechanism.

Contribution

It formulates the inverse VQA task, proposes a multi-modal dynamic inference model with attention adjustment, and introduces a new ranking metric for evaluation.

Findings

The model generates diverse and grammatically correct questions.

The proposed ranking metric effectively evaluates question relevance.

Experimental results demonstrate the model's ability to produce content-related questions.

Abstract

We propose the inverse problem of Visual question answering (iVQA), and explore its suitability as a benchmark for visuo-linguistic understanding. The iVQA task is to generate a question that corresponds to a given image and answer pair. Since the answers are less informative than the questions, and the questions have less learnable bias, an iVQA model needs to better understand the image to be successful than a VQA model. We pose question generation as a multi-modal dynamic inference process and propose an iVQA model that can gradually adjust its focus of attention guided by both a partially generated question and the answer. For evaluation, apart from existing linguistic metrics, we propose a new ranking metric. This metric compares the ground truth question's rank among a list of distractors, which allows the drawbacks of different algorithms and sources of error to be studied. Experimental results show that our model can generate diverse, grammatically correct and content correlated questions that match the given answer.