From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulation

TL;DR

This paper introduces FSD, a vision-language model that enhances robotic manipulation by reasoning about spatial relationships, leading to improved zero-shot generalization and success rates in various benchmarks and real-world tasks.

Contribution

FSD is a novel model that integrates spatial reasoning with vision-language understanding to improve robotic manipulation and generalization in unseen scenarios.

Findings

FSD achieves 40.6% success in SimplerEnv.

FSD attains 72% success across 8 real-world tasks.

Outperforms baseline methods by 30% in success rate.

Abstract

Achieving generalization in robotic manipulation remains a critical challenge, particularly for unseen scenarios and novel tasks. Current Vision-Language-Action (VLA) models, while building on top of general Vision-Language Models (VLMs), still fall short of achieving robust zero-shot performance due to the scarcity and heterogeneity prevalent in embodied datasets. To address these limitations, we propose FSD (From Seeing to Doing), a novel vision-language model that generates intermediate representations through spatial relationship reasoning, providing fine-grained guidance for robotic manipulation. Our approach combines a hierarchical data pipeline for training with a self-consistency mechanism that aligns spatial coordinates with visual signals. Through extensive experiments, we comprehensively validated FSD's capabilities in both "seeing" and "doing," achieving outstanding performance across 8 benchmarks for general spatial reasoning and embodied reference abilities, as well as on our proposed more challenging benchmark VABench. We also verified zero-shot capabilities in robot manipulation, demonstrating significant performance improvements over baseline methods in both SimplerEnv and real robot settings. Experimental results show that FSD achieves 40.6% success rate in SimplerEnv and 72% success rate across 8 real-world tasks, outperforming the strongest baseline by 30%.