SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning

TL;DR

SayPlan introduces a scalable method for grounding large language model-based robot task plans in complex 3D environments using hierarchical scene graphs, path planning, and iterative replanning to improve feasibility and execution.

Contribution

The paper presents a novel approach combining 3D scene graphs, hierarchical semantic search, classical path planning, and iterative replanning to enable large-scale, long-horizon robot task planning with LLMs.

Findings

Successfully plans and executes tasks in environments with up to 3 floors and 36 rooms.

Grounds large-scale, natural language instructions for mobile manipulators.

Demonstrates real robot execution with improved plan feasibility.

Abstract

Large language models (LLMs) have demonstrated impressive results in developing generalist planning agents for diverse tasks. However, grounding these plans in expansive, multi-floor, and multi-room environments presents a significant challenge for robotics. We introduce SayPlan, a scalable approach to LLM-based, large-scale task planning for robotics using 3D scene graph (3DSG) representations. To ensure the scalability of our approach, we: (1) exploit the hierarchical nature of 3DSGs to allow LLMs to conduct a 'semantic search' for task-relevant subgraphs from a smaller, collapsed representation of the full graph; (2) reduce the planning horizon for the LLM by integrating a classical path planner and (3) introduce an 'iterative replanning' pipeline that refines the initial plan using feedback from a scene graph simulator, correcting infeasible actions and avoiding planning failures. We evaluate our approach on two large-scale environments spanning up to 3 floors and 36 rooms with 140 assets and objects and show that our approach is capable of grounding large-scale, long-horizon task plans from abstract, and natural language instruction for a mobile manipulator robot to execute. We provide real robot video demonstrations on our project page https://sayplan.github.io.