Spatially Grounded Long-Horizon Task Planning in the Wild

TL;DR

This paper introduces GroundedPlanBench, a new benchmark for evaluating spatially grounded long-horizon task planning in robotics, and proposes V2GP, a framework leveraging real robot videos to enhance planning and grounding capabilities.

Contribution

The paper presents a novel benchmark for spatially grounded planning and a data generation framework that improves robot manipulation planning using real-world videos.

Findings

Spatially grounded planning is a major bottleneck for current VLMs.

V2GP improves action planning and spatial grounding performance.

Validated on benchmark and real-world robot experiments.

Abstract

Recent advances in robot manipulation increasingly leverage Vision-Language Models (VLMs) for high-level reasoning, such as decomposing task instructions into sequential action plans expressed in natural language that guide downstream low-level motor execution. However, current benchmarks do not assess whether these plans are spatially executable, particularly in specifying the exact spatial locations where the robot should interact to execute the plan, limiting evaluation of real-world manipulation capability. To bridge this gap, we define a novel task of grounded planning and introduce GroundedPlanBench, a newly curated benchmark for spatially grounded long-horizon action planning in the wild. GroundedPlanBench jointly evaluates hierarchical sub-action planning and spatial action grounding (where to act), enabling systematic assessment of whether generated sub-actions are spatially executable for robot manipulation. We further introduce Video-to-Spatially Grounded Planning (V2GP), an automated data generation framework that leverages real-world robot video demonstrations to improve spatially grounded long-horizon planning. Our evaluations reveal that spatially grounded long-horizon planning remains a major bottleneck for current VLMs. Our results demonstrate that V2GP provides a promising approach for improving both action planning and spatial grounding performance, validated on our benchmark as well as through real-world robot manipulation experiments, advancing progress toward spatially actionable planning.