RoboTracer: Mastering Spatial Trace with Reasoning in Vision-Language Models for Robotics

TL;DR

RoboTracer is a novel 3D-aware vision-language model designed for robotic spatial reasoning, capable of multi-step metric-grounded reasoning and spatial measurement, trained on a large-scale dataset and benchmarked with state-of-the-art performance.

Contribution

The paper introduces RoboTracer, a 3D-aware VLM with a universal spatial encoder and reinforcement fine-tuning, along with the TraceSpatial dataset and benchmark for complex spatial reasoning tasks.

Findings

RoboTracer achieves 79.1% success rate in spatial understanding tasks.

It surpasses previous models by 36% accuracy on TraceSpatial-Bench.

It can be integrated with control policies for real-world robotic tasks.

Abstract

Spatial tracing, as a fundamental embodied interaction ability for robots, is inherently challenging as it requires multi-step metric-grounded reasoning compounded with complex spatial referring and real-world metric measurement. However, existing methods struggle with this compositional task. To this end, we propose RoboTracer, a 3D-aware VLM that first achieves both 3D spatial referring and measuring via a universal spatial encoder and a regression-supervised decoder to enhance scale awareness during supervised fine-tuning (SFT). Moreover, RoboTracer advances multi-step metric-grounded reasoning via reinforcement fine-tuning (RFT) with metric-sensitive process rewards, supervising key intermediate perceptual cues to accurately generate spatial traces. To support SFT and RFT training, we introduce TraceSpatial, a large-scale dataset of 30M QA pairs, spanning outdoor/indoor/tabletop scenes and supporting complex reasoning processes (up to 9 steps). We further present TraceSpatial-Bench, a challenging benchmark filling the gap to evaluate spatial tracing. Experimental results show that RoboTracer surpasses baselines in spatial understanding, measuring, and referring, with an average success rate of 79.1%, and also achieves SOTA performance on TraceSpatial-Bench by a large margin, exceeding Gemini-2.5-Pro by 36% accuracy. Notably, RoboTracer can be integrated with various control policies to execute long-horizon, dynamic tasks across diverse robots (UR5, G1 humanoid) in cluttered real-world scenes. See the project page at https://zhoues.github.io/RoboTracer.