RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics

TL;DR

RoboSpatial introduces a large-scale dataset with rich spatial annotations from real indoor scenes to enhance the spatial reasoning capabilities of vision-language models in robotics, addressing current limitations in understanding reference frames.

Contribution

The paper presents RoboSpatial, a comprehensive dataset combining 2D images and 3D scans with detailed spatial annotations to improve robotic spatial understanding in vision-language models.

Findings

Models trained on RoboSpatial outperform baselines in spatial reasoning tasks.

The dataset enables better understanding of reference frames in robotic perception.

Enhanced spatial reasoning improves robot manipulation capabilities.

Abstract

Spatial understanding is a crucial capability that enables robots to perceive their surroundings, reason about their environment, and interact with it meaningfully. In modern robotics, these capabilities are increasingly provided by vision-language models. However, these models face significant challenges in spatial reasoning tasks, as their training data are based on general-purpose image datasets that often lack sophisticated spatial understanding. For example, datasets frequently do not capture reference frame comprehension, yet effective spatial reasoning requires understanding whether to reason from ego-, world-, or object-centric perspectives. To address this issue, we introduce RoboSpatial, a large-scale dataset for spatial understanding in robotics. It consists of real indoor and tabletop scenes, captured as 3D scans and egocentric images, and annotated with rich spatial information relevant to robotics. The dataset includes 1M images, 5k 3D scans, and 3M annotated spatial relationships, and the pairing of 2D egocentric images with 3D scans makes it both 2D- and 3D- ready. Our experiments show that models trained with RoboSpatial outperform baselines on downstream tasks such as spatial affordance prediction, spatial relationship prediction, and robot manipulation.