VQA-MHUG: A Gaze Dataset to Study Multimodal Neural Attention in Visual Question Answering

TL;DR

This paper introduces VQA-MHUG, a new gaze dataset capturing human attention during visual question answering, and analyzes how neural models' attention strategies compare to humans, revealing that text attention correlates with model performance.

Contribution

The paper presents a novel multimodal gaze dataset for VQA and provides the first analysis linking human and neural attention strategies, highlighting the importance of text attention.

Findings

Higher correlation with human text attention predicts better VQA performance

Neural models show similar attention patterns to humans on text modality

Insights suggest improving neural text attention could enhance VQA models

Abstract

We present VQA-MHUG - a novel 49-participant dataset of multimodal human gaze on both images and questions during visual question answering (VQA) collected using a high-speed eye tracker. We use our dataset to analyze the similarity between human and neural attentive strategies learned by five state-of-the-art VQA models: Modular Co-Attention Network (MCAN) with either grid or region features, Pythia, Bilinear Attention Network (BAN), and the Multimodal Factorized Bilinear Pooling Network (MFB). While prior work has focused on studying the image modality, our analyses show - for the first time - that for all models, higher correlation with human attention on text is a significant predictor of VQA performance. This finding points at a potential for improving VQA performance and, at the same time, calls for further research on neural text attention mechanisms and their integration into architectures for vision and language tasks, including but potentially also beyond VQA.