On the Flip Side: Identifying Counterexamples in Visual Question Answering

TL;DR

This paper investigates whether current VQA models truly understand key semantic distinctions by testing their ability to identify counterexamples, revealing limitations in their visual-semantic reasoning capabilities.

Contribution

The paper introduces a new counterexample prediction task, VQA-CX, and evaluates existing models, highlighting their limited understanding of semantic distinctions despite strong benchmark performance.

Findings

Models outperform existing benchmarks on VQA-CX.

State-of-the-art VQA models' representations do not significantly aid counterexample prediction.

Performance on VQA benchmarks may not reflect true semantic reasoning abilities.

Abstract

Visual question answering (VQA) models respond to open-ended natural language questions about images. While VQA is an increasingly popular area of research, it is unclear to what extent current VQA architectures learn key semantic distinctions between visually-similar images. To investigate this question, we explore a reformulation of the VQA task that challenges models to identify counterexamples: images that result in a different answer to the original question. We introduce two methods for evaluating existing VQA models against a supervised counterexample prediction task, VQA-CX. While our models surpass existing benchmarks on VQA-CX, we find that the multimodal representations learned by an existing state-of-the-art VQA model do not meaningfully contribute to performance on this task. These results call into question the assumption that successful performance on the VQA benchmark is indicative of general visual-semantic reasoning abilities.