Retrieval-Augmented Hierarchical in-Context Reinforcement Learning and Hindsight Modular Reflections for Task Planning with LLMs

TL;DR

This paper introduces RAHL, a framework combining hierarchical reinforcement learning with retrieval-augmented in-context learning, enabling LLMs to decompose tasks and improve decision-making in robotics and benchmark environments.

Contribution

The paper presents a novel RAHL framework that integrates LLM-based task decomposition with hindsight modular reflection for improved multi-episode decision-making.

Findings

RAHL improves performance by up to 42% in benchmark tasks.

Hindsight Modular Reflection enhances reflection efficiency on sub-trajectories.

RAHL successfully controls a robot to navigate and find environments.

Abstract

Large Language Models (LLMs) have demonstrated remarkable abilities in various language tasks, making them promising candidates for decision-making in robotics. Inspired by Hierarchical Reinforcement Learning (HRL), we propose Retrieval-Augmented in-context reinforcement Learning (RAHL), a novel framework that decomposes complex tasks into sub-tasks using an LLM-based high-level policy, in which a complex task is decomposed into sub-tasks by a high-level policy on-the-fly. The sub-tasks, defined by goals, are assigned to the low-level policy to complete. To improve the agent's performance in multi-episode execution, we propose Hindsight Modular Reflection (HMR), where, instead of reflecting on the full trajectory, we let the agent reflect on shorter sub-trajectories to improve reflection efficiency. We evaluated the decision-making ability of the proposed RAHL in three benchmark environments--ALFWorld, Webshop, and HotpotQA. The results show that RAHL can achieve an improvement in performance in 9%, 42%, and 10% in 5 episodes of execution in strong baselines. Furthermore, we also implemented RAHL on the Boston Dynamics SPOT robot. The experiment shows that the robot can scan the environment, find entrances, and navigate to new rooms controlled by the LLM policy.