Integrated Exploration and Sequential Manipulation on Scene Graph with LLM-based Situated Replanning

TL;DR

This paper introduces EPoG, a novel framework combining exploration and task planning using scene graphs and LLMs, enabling robots to efficiently manipulate objects in partially known environments.

Contribution

The paper presents EPoG, integrating a graph-based global planner with an LLM-based local planner for continuous environment understanding and manipulation planning.

Findings

Achieved 91.3% success rate in household scenes.

Reduced travel distance by 36.1% on average.

Successfully deployed on a physical robot in dynamic environments.

Abstract

In partially known environments, robots must combine exploration to gather information with task planning for efficient execution. To address this challenge, we propose EPoG, an Exploration-based sequential manipulation Planning framework on Scene Graphs. EPoG integrates a graph-based global planner with a Large Language Model (LLM)-based situated local planner, continuously updating a belief graph using observations and LLM predictions to represent known and unknown objects. Action sequences are generated by computing graph edit operations between the goal and belief graphs, ordered by temporal dependencies and movement costs. This approach seamlessly combines exploration and sequential manipulation planning. In ablation studies across 46 realistic household scenes and 5 long-horizon daily object transportation tasks, EPoG achieved a success rate of 91.3%, reducing travel distance by 36.1% on average. Furthermore, a physical mobile manipulator successfully executed complex tasks in unknown and dynamic environments, demonstrating EPoG's potential for real-world applications.