An LLM-powered Natural-to-Robotic Language Translation Framework with Correctness Guarantees

TL;DR

This paper presents a framework that translates natural language to robot control programs using LLMs, with correctness verification and feedback-based fine-tuning to improve accuracy and reliability in robotic applications.

Contribution

It introduces a Robot Skill Language (RSL) and a compiler/debugger system that verify and refine LLM-generated control programs, ensuring correctness before execution.

Findings

NRTrans outperforms existing methods across various LLMs and tasks.

High success rate achieved with lightweight LLMs.

Provides correctness guarantees for robot control programs.

Abstract

The Large Language Models (LLM) are increasingly being deployed in robotics to generate robot control programs for specific user tasks, enabling embodied intelligence. Existing methods primarily focus on LLM training and prompt design that utilize LLMs to generate executable programs directly from user tasks in natural language. However, due to the inconsistency of the LLMs and the high complexity of the tasks, such best-effort approaches often lead to tremendous programming errors in the generated code, which significantly undermines the effectiveness especially when the light-weight LLMs are applied. This paper introduces a natural-robotic language translation framework that (i) provides correctness verification for generated control programs and (ii) enhances the performance of LLMs in program generation via feedback-based fine-tuning for the programs. To achieve this, a Robot Skill Language (RSL) is proposed to abstract away from the intricate details of the control programs, bridging the natural language tasks with the underlying robot skills. Then, the RSL compiler and debugger are constructed to verify RSL programs generated by the LLM and provide error feedback to the LLM for refining the outputs until being verified by the compiler. This provides correctness guarantees for the LLM-generated programs before being offloaded to the robots for execution, significantly enhancing the effectiveness of LLM-powered robotic applications. Experiments demonstrate NRTrans outperforms the existing method under a range of LLMs and tasks, and achieves a high success rate for light-weight LLMs.