Grounding Language Models in Autonomous Loco-manipulation Tasks

TL;DR

This paper introduces a framework combining reinforcement learning, whole-body optimization, and large language models to enable humanoid robots to perform complex, open-ended loco-manipulation tasks with high autonomy in unstructured environments.

Contribution

It presents a novel hierarchical planning approach that integrates language models with motion primitives for versatile humanoid robot control.

Findings

Effective adaptation to new loco-manipulation tasks in simulation and real-world

High autonomy demonstrated from free-text commands in unstructured scenes

Successful integration of LLMs with motion planning for complex tasks

Abstract

Humanoid robots with behavioral autonomy have consistently been regarded as ideal collaborators in our daily lives and promising representations of embodied intelligence. Compared to fixed-based robotic arms, humanoid robots offer a larger operational space while significantly increasing the difficulty of control and planning. Despite the rapid progress towards general-purpose humanoid robots, most studies remain focused on locomotion ability with few investigations into whole-body coordination and tasks planning, thus limiting the potential to demonstrate long-horizon tasks involving both mobility and manipulation under open-ended verbal instructions. In this work, we propose a novel framework that learns, selects, and plans behaviors based on tasks in different scenarios. We combine reinforcement learning (RL) with whole-body optimization to generate robot motions and store them into a motion library. We further leverage the planning and reasoning features of the large language model (LLM), constructing a hierarchical task graph that comprises a series of motion primitives to bridge lower-level execution with higher-level planning. Experiments in simulation and real-world using the CENTAURO robot show that the language model based planner can efficiently adapt to new loco-manipulation tasks, demonstrating high autonomy from free-text commands in unstructured scenes.