LLM-GROP: Visually Grounded Robot Task and Motion Planning with Large Language Models

TL;DR

This paper introduces LLM-GROP, a framework that combines large language models and computer vision to improve task and motion planning for mobile manipulation tasks involving multiple objects, especially in complex, real-world scenarios.

Contribution

It presents a novel TAMP framework leveraging LLMs and vision to handle common sense reasoning and adaptive planning in multi-object rearrangement tasks.

Findings

Achieved 84.4% success rate in real-world object rearrangement tasks.

Demonstrated effective integration of LLMs and vision for adaptive planning.

Showed potential for improved robot performance in complex environments.

Abstract

Task planning and motion planning are two of the most important problems in robotics, where task planning methods help robots achieve high-level goals and motion planning methods maintain low-level feasibility. Task and motion planning (TAMP) methods interleave the two processes of task planning and motion planning to ensure goal achievement and motion feasibility. Within the TAMP context, we are concerned with the mobile manipulation (MoMa) of multiple objects, where it is necessary to interleave actions for navigation and manipulation. In particular, we aim to compute where and how each object should be placed given underspecified goals, such as ``set up dinner table with a fork, knife and plate.'' We leverage the rich common sense knowledge from large language models (LLMs), e.g., about how tableware is organized, to facilitate both task-level and motion-level planning. In addition, we use computer vision methods to learn a strategy for selecting base positions to facilitate MoMa behaviors, where the base position corresponds to the robot's ``footprint'' and orientation in its operating space. Altogether, this article provides a principled TAMP framework for MoMa tasks that accounts for common sense about object rearrangement and is adaptive to novel situations that include many objects that need to be moved. We performed quantitative experiments in both real-world settings and simulated environments. We evaluated the success rate and efficiency in completing long-horizon object rearrangement tasks. While the robot completed 84.4\% real-world object rearrangement trials, subjective human evaluations indicated that the robot's performance is still lower than experienced human waiters.