On the Cognition of Visual Question Answering Models and Human Intelligence: A Comparative Study

TL;DR

This paper compares VQA models to human cognition by analyzing their outputs and attention maps, revealing similarities in recognition but differences in reasoning, and suggests directions for enhancing cognitive capabilities in models.

Contribution

It introduces a survey-based analysis comparing VQA models and humans, highlighting their similarities and differences in cognition and reasoning.

Findings

VQA models resemble humans in architecture and recognition performance.

Models struggle with cognitive inference tasks.

Analysis guides future research to improve cognitive capacity in models.

Abstract

Visual Question Answering (VQA) is a challenging task that requires cross-modal understanding and reasoning of visual image and natural language question. To inspect the association of VQA models to human cognition, we designed a survey to record human thinking process and analyzed VQA models by comparing the outputs and attention maps with those of humans. We found that although the VQA models resemble human cognition in architecture and performs similarly with human on the recognition-level, they still struggle with cognitive inferences. The analysis of human thinking procedure serves to direct future research and introduce more cognitive capacity into modeling features and architectures.