LLM Augmented Hierarchical Agents

TL;DR

This paper introduces a hierarchical agent that combines LLMs for planning with reinforcement learning for environment interaction, enabling efficient learning of long-horizon tasks in simulation and real-world settings without relying on LLMs during deployment.

Contribution

The paper presents a novel approach that leverages LLMs for high-level planning within a hierarchical RL framework, improving sample efficiency and task performance.

Findings

Outperforms baseline methods in simulation environments.

Achieves successful real-world block manipulation with a robot arm.

Does not require LLMs during deployment after training.

Abstract

Solving long-horizon, temporally-extended tasks using Reinforcement Learning (RL) is challenging, compounded by the common practice of learning without prior knowledge (or tabula rasa learning). Humans can generate and execute plans with temporally-extended actions and quickly learn to perform new tasks because we almost never solve problems from scratch. We want autonomous agents to have this same ability. Recently, LLMs have been shown to encode a tremendous amount of knowledge about the world and to perform impressive in-context learning and reasoning. However, using LLMs to solve real world problems is hard because they are not grounded in the current task. In this paper we exploit the planning capabilities of LLMs while using RL to provide learning from the environment, resulting in a hierarchical agent that uses LLMs to solve long-horizon tasks. Instead of completely relying on LLMs, they guide a high-level policy, making learning significantly more sample efficient. This approach is evaluated in simulation environments such as MiniGrid, SkillHack, and Crafter, and on a real robot arm in block manipulation tasks. We show that agents trained using our approach outperform other baselines methods and, once trained, don't need access to LLMs during deployment.