What is needed for simple spatial language capabilities in VQA?

TL;DR

This paper investigates the core components needed for simple spatial language understanding in visual question answering models, analyzing various models and modifications to improve spatial reasoning capabilities.

Contribution

It provides a comparative analysis of models on spatial relation tasks and identifies key factors for enhancing spatial language abilities in VQA systems.

Findings

Certain model modifications significantly improve spatial reasoning.

Model performance varies notably across different spatial relation types.

Insights into the minimal requirements for effective spatial language understanding.

Abstract

Visual question answering (VQA) comprises a variety of language capabilities. The diagnostic benchmark dataset CLEVR has fueled progress by helping to better assess and distinguish models in basic abilities like counting, comparing and spatial reasoning in vitro. Following this approach, we focus on spatial language capabilities and investigate the question: what are the key ingredients to handle simple visual-spatial relations? We look at the SAN, RelNet, FiLM and MC models and evaluate their learning behavior on diagnostic data which is solely focused on spatial relations. Via comparative analysis and targeted model modification we identify what really is required to substantially improve upon the CNN-LSTM baseline.