Interpretable Visual Question Answering via Reasoning Supervision

TL;DR

This paper introduces a novel VQA model that uses reasoning supervision from textual justifications to improve visual grounding and performance without needing explicit grounding annotations.

Contribution

The work proposes a new architecture that leverages reasoning supervision from textual justifications to enhance visual grounding in VQA models.

Findings

Improved visual perception and reasoning in VQA models.

Enhanced performance on VQA tasks without explicit grounding annotations.

Qualitative evidence of better visual attention alignment.

Abstract

Transformer-based architectures have recently demonstrated remarkable performance in the Visual Question Answering (VQA) task. However, such models are likely to disregard crucial visual cues and often rely on multimodal shortcuts and inherent biases of the language modality to predict the correct answer, a phenomenon commonly referred to as lack of visual grounding. In this work, we alleviate this shortcoming through a novel architecture for visual question answering that leverages common sense reasoning as a supervisory signal. Reasoning supervision takes the form of a textual justification of the correct answer, with such annotations being already available on large-scale Visual Common Sense Reasoning (VCR) datasets. The model's visual attention is guided toward important elements of the scene through a similarity loss that aligns the learned attention distributions guided by the question and the correct reasoning. We demonstrate both quantitatively and qualitatively that the proposed approach can boost the model's visual perception capability and lead to performance increase, without requiring training on explicit grounding annotations.