Knowing Earlier what Right Means to You: A Comprehensive VQA Dataset for Grounding Relative Directions via Multi-Task Learning

TL;DR

This paper introduces GRiD-A-3D, a new VQA dataset for grounding relative directions, enabling detailed analysis of models' spatial reasoning with fewer resources, and shows models learn reasoning subtasks in an intuitive order.

Contribution

The paper presents GRiD-A-3D, a novel dataset for evaluating spatial reasoning in VQA, along with an analysis of model learning dynamics on this dataset.

Findings

Models quickly learn sub-tasks like object recognition and orientation estimation.

Training on GRiD-A-3D requires less computational resources.

Models develop an intuitive reasoning order over epochs.

Abstract

Spatial reasoning poses a particular challenge for intelligent agents and is at the same time a prerequisite for their successful interaction and communication in the physical world. One such reasoning task is to describe the position of a target object with respect to the intrinsic orientation of some reference object via relative directions. In this paper, we introduce GRiD-A-3D, a novel diagnostic visual question-answering (VQA) dataset based on abstract objects. Our dataset allows for a fine-grained analysis of end-to-end VQA models' capabilities to ground relative directions. At the same time, model training requires considerably fewer computational resources compared with existing datasets, yet yields a comparable or even higher performance. Along with the new dataset, we provide a thorough evaluation based on two widely known end-to-end VQA architectures trained on GRiD-A-3D. We demonstrate that within a few epochs, the subtasks required to reason over relative directions, such as recognizing and locating objects in a scene and estimating their intrinsic orientations, are learned in the order in which relative directions are intuitively processed.