Human Attention in Visual Question Answering: Do Humans and Deep Networks Look at the Same Regions?

TL;DR

This study investigates whether deep learning models in Visual Question Answering focus on the same image regions as humans do, revealing significant differences in attention patterns.

Contribution

We introduce the VQA-HAT dataset with human attention annotations and evaluate how current models' attention compares to human attention in VQA tasks.

Findings

Current models do not align with human attention patterns.

Human and model attention regions show low correlation.

New dataset enables better understanding of attention in VQA.

Abstract

We conduct large-scale studies on `human attention' in Visual Question Answering (VQA) to understand where humans choose to look to answer questions about images. We design and test multiple game-inspired novel attention-annotation interfaces that require the subject to sharpen regions of a blurred image to answer a question. Thus, we introduce the VQA-HAT (Human ATtention) dataset. We evaluate attention maps generated by state-of-the-art VQA models against human attention both qualitatively (via visualizations) and quantitatively (via rank-order correlation). Overall, our experiments show that current attention models in VQA do not seem to be looking at the same regions as humans.