Human Attention in Visual Question Answering: Do Humans and Deep Networks Look at the Same Regions?

TL;DR

This study investigates whether deep VQA models focus on the same image regions as humans by creating a new dataset and comparing model attention maps to human attention, revealing significant differences.

Contribution

Introduces the VQA-HAT dataset with human attention annotations and evaluates the alignment between human and model attention in VQA tasks.

Findings

Current VQA attention models do not match human attention patterns.

Humans tend to focus on different image regions than models during VQA.

The dataset enables better understanding of attention discrepancies in VQA.

Abstract

We conduct large-scale studies on `human attention' in Visual Question Answering (VQA) to understand where humans choose to look to answer questions about images. We design and test multiple game-inspired novel attention-annotation interfaces that require the subject to sharpen regions of a blurred image to answer a question. Thus, we introduce the VQA-HAT (Human ATtention) dataset. We evaluate attention maps generated by state-of-the-art VQA models against human attention both qualitatively (via visualizations) and quantitatively (via rank-order correlation). Overall, our experiments show that current attention models in VQA do not seem to be looking at the same regions as humans.