SPEAR-1: Scaling Beyond Robot Demonstrations via 3D Understanding

TL;DR

This paper introduces SPEAR-1, a robotic foundation model that combines 3D understanding with language-instructed control, trained on large-scale non-robotic data to improve generalization and reduce reliance on robot demonstrations.

Contribution

The authors develop SPEAR-VLM, a 3D-aware vision-language model, and integrate it into SPEAR-1, achieving superior embodied control with fewer robot demonstrations.

Findings

SPEAR-1 outperforms state-of-the-art models like π₀-FAST and π₀.5.

It uses 20 times fewer robot demonstrations.

The model demonstrates improved reliability in embodied control.

Abstract

Robotic Foundation Models (RFMs) hold great promise as generalist, end-to-end systems for robot control. Yet their ability to generalize across new environments, tasks, and embodiments remains limited. We argue that a major bottleneck lies in their foundations: most RFMs are built by fine-tuning internet-pretrained Vision-Language Models (VLMs). However, these VLMs are trained on 2D image-language tasks and lack the 3D spatial reasoning inherently required for embodied control in the 3D world. Bridging this gap directly with large-scale robotic data is costly and difficult to scale. Instead, we propose to enrich easy-to-collect non-robotic image data with 3D annotations and enhance a pretrained VLM with 3D understanding capabilities. Following this strategy, we train SPEAR-VLM, a 3D-aware VLM that infers object coordinates in 3D space from a single 2D image. Building on SPEAR-VLM, we introduce our main contribution, $~\textbf{SPEAR-1}$: a robotic foundation model that integrates grounded 3D perception with language-instructed embodied control. Trained on $\sim$45M frames from 24 Open X-Embodiment datasets, SPEAR-1 outperforms or matches state-of-the-art models such as $\pi_0$-FAST and $\pi_{0.5}$, while it uses 20$\times$ fewer robot demonstrations. This carefully-engineered training strategy unlocks new VLM capabilities and as a consequence boosts the reliability of embodied control beyond what is achievable with only robotic data. We make our model weights and 3D-annotated datasets publicly available at https://spear.insait.ai.