NovaPlan: Zero-Shot Long-Horizon Manipulation via Closed-Loop Video Language Planning

TL;DR

NovaPlan is a hierarchical, zero-shot framework that combines vision-language models and video planning with geometric grounding to enable robots to perform complex long-horizon tasks with error recovery.

Contribution

It introduces a novel closed-loop, hierarchical approach integrating VLMs and video generation for zero-shot long-horizon manipulation with autonomous re-planning.

Findings

Effective in complex assembly tasks

Demonstrates robust error recovery behaviors

Achieves stable execution under occlusion and depth inaccuracies

Abstract

Solving long-horizon tasks requires robots to integrate high-level semantic reasoning with low-level physical interaction. While vision-language models (VLMs) and video generation models can decompose tasks and imagine outcomes, they often lack the physical grounding necessary for real-world execution. We introduce NovaPlan, a hierarchical framework that unifies closed-loop VLM and video planning with geometrically grounded robot execution for zero-shot long-horizon manipulation. At the high level, a VLM planner decomposes tasks into sub-goals and monitors robot execution in a closed loop, enabling the system to recover from single-step failures through autonomous re-planning. To compute low-level robot actions, we extract and utilize both task-relevant object keypoints and human hand poses as kinematic priors from the generated videos, and employ a switching mechanism to choose the better one as a reference for robot actions, maintaining stable execution even under heavy occlusion or depth inaccuracy. We demonstrate the effectiveness of NovaPlan on three long-horizon tasks and the Functional Manipulation Benchmark (FMB). Our results show that NovaPlan can perform complex assembly tasks and exhibit dexterous error recovery behaviors without any prior demonstrations or training. Project page: https://nova-plan.github.io/