VQA Therapy: Exploring Answer Differences by Visually Grounding Answers

TL;DR

This paper introduces VQAAnswerTherapy, a dataset that visually grounds each unique answer to a visual question, and explores problems of predicting answer grounding consistency and localization, revealing strengths and weaknesses of current algorithms.

Contribution

It presents the first dataset with answer groundings for visual questions and proposes new problems for predicting and localizing answer groundings.

Findings

Modern algorithms vary in success across tasks

The dataset reveals specific challenges in grounding answers

Benchmark results highlight areas for improvement

Abstract

Visual question answering is a task of predicting the answer to a question about an image. Given that different people can provide different answers to a visual question, we aim to better understand why with answer groundings. We introduce the first dataset that visually grounds each unique answer to each visual question, which we call VQAAnswerTherapy. We then propose two novel problems of predicting whether a visual question has a single answer grounding and localizing all answer groundings. We benchmark modern algorithms for these novel problems to show where they succeed and struggle. The dataset and evaluation server can be found publicly at https://vizwiz.org/tasks-and-datasets/vqa-answer-therapy/.