Action Contextualization: Adaptive Task Planning and Action Tuning using Large Language Models

TL;DR

This paper presents a framework that enhances robot adaptability and error correction in task planning by integrating large language models with motion evaluation and online feedback, demonstrated through autonomous table-clearing tasks.

Contribution

It introduces a novel action contextualization framework that tailors robot actions based on context, integrating motion metrics and online feedback for improved adaptability and autonomy.

Findings

Achieved an 81.25% success rate in experimental validation.

Demonstrated autonomous error correction and robustness against disturbances.

Enabled seamless integration with dynamical system-based controllers.

Abstract

Large Language Models (LLMs) present a promising frontier in robotic task planning by leveraging extensive human knowledge. Nevertheless, the current literature often overlooks the critical aspects of robots' adaptability and error correction. This work aims to overcome this limitation by enabling robots to modify their motions and select the most suitable task plans based on the context. We introduce a novel framework to achieve action contextualization, aimed at tailoring robot actions to the context of specific tasks, thereby enhancing adaptability through applying LLM-derived contextual insights. Our framework integrates motion metrics that evaluate robot performances for each motion to resolve redundancy in planning. Moreover, it supports online feedback between the robot and the LLM, enabling immediate modifications to the task plans and corrections of errors. An overall success rate of 81.25% has been achieved through extensive experimental validation. Finally, when integrated with dynamical system (DS)-based robot controllers, the robotic arm-hand system demonstrates its proficiency in autonomously executing LLM-generated motion plans for sequential table-clearing tasks, rectifying errors without human intervention, and showcasing robustness against external disturbances. Our proposed framework also features the potential to be integrated with modular control approaches, significantly enhancing robots' adaptability and autonomy in performing sequential tasks in the real world.