Accelerating Reinforcement Learning of Robotic Manipulations via Feedback from Large Language Models

TL;DR

This paper presents Lafite-RL, a framework that leverages large language models to provide feedback for reinforcement learning in robotic manipulation, significantly improving learning efficiency and success rates.

Contribution

Introducing Lafite-RL, a novel framework that uses LLMs to guide RL agents in robotic tasks through natural language feedback, enhancing sample efficiency and performance.

Findings

Lafite-RL outperforms baseline methods in RLBench tasks.

LLM-guided RL improves learning efficiency and success rate.

Simple natural language prompts effectively guide robotic learning.

Abstract

Reinforcement Learning (RL) plays an important role in the robotic manipulation domain since it allows self-learning from trial-and-error interactions with the environment. Still, sample efficiency and reward specification seriously limit its potential. One possible solution involves learning from expert guidance. However, obtaining a human expert is impractical due to the high cost of supervising an RL agent, and developing an automatic supervisor is a challenging endeavor. Large Language Models (LLMs) demonstrate remarkable abilities to provide human-like feedback on user inputs in natural language. Nevertheless, they are not designed to directly control low-level robotic motions, as their pretraining is based on vast internet data rather than specific robotics data. In this paper, we introduce the Lafite-RL (Language agent feedback interactive Reinforcement Learning) framework, which enables RL agents to learn robotic tasks efficiently by taking advantage of LLMs' timely feedback. Our experiments conducted on RLBench tasks illustrate that, with simple prompt design in natural language, the Lafite-RL agent exhibits improved learning capabilities when guided by an LLM. It outperforms the baseline in terms of both learning efficiency and success rate, underscoring the efficacy of the rewards provided by an LLM.