A Spatial Relationship Aware Dataset for Robotics

TL;DR

This paper introduces a new dataset of indoor robot images annotated with detailed spatial relationships, benchmarking models and demonstrating improved spatial reasoning for robotic planning.

Contribution

It provides a novel spatial-relationship-aware dataset and shows how explicit spatial information enhances foundation models' robotic planning capabilities.

Findings

Benchmarking reveals significant performance differences among models.

Explicit spatial relationships improve plan generation accuracy.

Dataset and tools are publicly available for further research.

Abstract

Robotic task planning in real-world environments requires not only object recognition but also a nuanced understanding of spatial relationships between objects. We present a spatial-relationship-aware dataset of nearly 1,000 robot-acquired indoor images, annotated with object attributes, positions, and detailed spatial relationships. Captured using a Boston Dynamics Spot robot and labelled with a custom annotation tool, the dataset reflects complex scenarios with similar or identical objects and intricate spatial arrangements. We benchmark six state-of-the-art scene-graph generation models on this dataset, analysing their inference speed and relational accuracy. Our results highlight significant differences in model performance and demonstrate that integrating explicit spatial relationships into foundation models, such as ChatGPT 4o, substantially improves their ability to generate executable, spatially-aware plans for robotics. The dataset and annotation tool are publicly available at https://github.com/PengPaulWang/SpatialAwareRobotDataset, supporting further research in spatial reasoning for robotics.