SPARE3D: A Dataset for SPAtial REasoning on Three-View Line Drawings

TL;DR

SPARE3D introduces a dataset for evaluating deep networks' ability to perform spatial reasoning on 2D line drawings of 3D objects, revealing current models' limitations compared to human performance.

Contribution

The paper presents the SPARE3D dataset with challenging 2D-3D reasoning tasks and a method for generating supervision data, enabling assessment of neural networks' spatial reasoning capabilities.

Findings

Deep networks underperform compared to humans on SPARE3D tasks.

Current models perform close to random guessing on complex spatial reasoning tasks.

The dataset facilitates future research in improving AI spatial reasoning abilities.

Abstract

Spatial reasoning is an important component of human intelligence. We can imagine the shapes of 3D objects and reason about their spatial relations by merely looking at their three-view line drawings in 2D, with different levels of competence. Can deep networks be trained to perform spatial reasoning tasks? How can we measure their "spatial intelligence"? To answer these questions, we present the SPARE3D dataset. Based on cognitive science and psychometrics, SPARE3D contains three types of 2D-3D reasoning tasks on view consistency, camera pose, and shape generation, with increasing difficulty. We then design a method to automatically generate a large number of challenging questions with ground truth answers for each task. They are used to provide supervision for training our baseline models using state-of-the-art architectures like ResNet. Our experiments show that although convolutional networks have achieved superhuman performance in many visual learning tasks, their spatial reasoning performance on SPARE3D tasks is either lower than average human performance or even close to random guesses. We hope SPARE3D can stimulate new problem formulations and network designs for spatial reasoning to empower intelligent robots to operate effectively in the 3D world via 2D sensors. The dataset and code are available at https://ai4ce.github.io/SPARE3D.