On the Role of Visual Grounding in VQA

TL;DR

This paper formalizes the role of visual grounding in VQA using a new theoretical framework, revealing how shortcut learning exploits VG and proposing improved OOD tests to better evaluate VG's importance.

Contribution

It introduces the Visually Grounded Reasoning (VGR) framework to formalize VG's role in VQA and proposes new OOD tests emphasizing VG requirement.

Findings

VG-related shortcut learning is prevalent in VQA models.

Current OOD tests may not effectively measure VG importance.

Proposed OOD tests improve VG evaluation and model performance.

Abstract

Visual Grounding (VG) in VQA refers to a model's proclivity to infer answers based on question-relevant image regions. Conceptually, VG identifies as an axiomatic requirement of the VQA task. In practice, however, DNN-based VQA models are notorious for bypassing VG by way of shortcut (SC) learning without suffering obvious performance losses in standard benchmarks. To uncover the impact of SC learning, Out-of-Distribution (OOD) tests have been proposed that expose a lack of VG with low accuracy. These tests have since been at the center of VG research and served as basis for various investigations into VG's impact on accuracy. However, the role of VG in VQA still remains not fully understood and has not yet been properly formalized. In this work, we seek to clarify VG's role in VQA by formalizing it on a conceptual level. We propose a novel theoretical framework called "Visually Grounded Reasoning" (VGR) that uses the concepts of VG and Reasoning to describe VQA inference in ideal OOD testing. By consolidating fundamental insights into VG's role in VQA, VGR helps to reveal rampant VG-related SC exploitation in OOD testing, which explains why the relationship between VG and OOD accuracy has been difficult to define. Finally, we propose an approach to create OOD tests that properly emphasize a requirement for VG, and show how to improve performance on them.