Natural Language as Policies: Reasoning for Coordinate-Level Embodied Control with LLMs

TL;DR

This paper presents a novel approach where large language models interpret natural language descriptions to directly generate coordinate-level control commands for robotics, bypassing traditional code-based policies.

Contribution

It introduces a natural language reasoning method for robotics task planning that reduces reliance on intermediate code representations and enhances transferability to unseen tasks.

Findings

Natural language reasoning improves task success rates.

The approach enables transfer of skills to new, unseen tasks.

Prompt engineering significantly boosts performance.

Abstract

We demonstrate experimental results with LLMs that address robotics task planning problems. Recently, LLMs have been applied in robotics task planning, particularly using a code generation approach that converts complex high-level instructions into mid-level policy codes. In contrast, our approach acquires text descriptions of the task and scene objects, then formulates task planning through natural language reasoning, and outputs coordinate level control commands, thus reducing the necessity for intermediate representation code as policies with pre-defined APIs. Our approach is evaluated on a multi-modal prompt simulation benchmark, demonstrating that our prompt engineering experiments with natural language reasoning significantly enhance success rates compared to its absence. Furthermore, our approach illustrates the potential for natural language descriptions to transfer robotics skills from known tasks to previously unseen tasks. The project website: https://natural-language-as-policies.github.io/