Ground Manipulator Primitive Tasks to Executable Actions using Large Language Models

TL;DR

This paper introduces a method using large language models to translate high-level robotic manipulation tasks into low-level actions, bridging the planning and execution layers in robot systems.

Contribution

It proposes a novel prompt design based on task frame formalism to enable LLMs to generate precise control set-points for robot manipulators.

Findings

LLMs can effectively generate position/force set-points for manipulation tasks.

The approach demonstrates promising results across several state-of-the-art LLMs.

The method simplifies the transition from planning to execution in robotic systems.

Abstract

Layered architectures have been widely used in robot systems. The majority of them implement planning and execution functions in separate layers. However, there still lacks a straightforward way to transit high-level tasks in the planning layer to the low-level motor commands in the execution layer. In order to tackle this challenge, we propose a novel approach to ground the manipulator primitive tasks to robot low-level actions using large language models (LLMs). We designed a program-function-like prompt based on the task frame formalism. In this way, we enable LLMs to generate position/force set-points for hybrid control. Evaluations over several state-of-the-art LLMs are provided.