SAC-GLAM: Improving Online RL for LLM agents with Soft Actor-Critic and Hindsight Relabeling

TL;DR

This paper introduces SAC-GLAM, an off-policy RL method with hindsight relabeling for LLM agents, enabling more efficient online learning and autonomous goal pursuit in complex environments.

Contribution

It adapts Soft Actor-Critic and hindsight relabeling for LLM agents, enhancing online learning capabilities beyond on-policy methods.

Findings

Outperforms on-policy methods in multi-goal RL environments.

Enables autonomous intrinsically motivated LLM agents.

Facilitates learning of efficient strategies in complex tasks.

Abstract

The past years have seen Large Language Models (LLMs) strive not only as generative models but also as agents solving textual sequential decision-making tasks. When facing complex environments where their zero-shot abilities are insufficient, recent work showed online Reinforcement Learning (RL) could be used for the LLM agent to discover and learn efficient strategies interactively. However, most prior work sticks to on-policy algorithms, which greatly reduces the scope of methods such agents could use for both exploration and exploitation, such as experience replay and hindsight relabeling. Yet, such methods may be key for LLM learning agents, and in particular when designing autonomous intrinsically motivated agents sampling and pursuing their own goals (i.e. autotelic agents). This paper presents and studies an adaptation of Soft Actor-Critic and hindsight relabeling to LLM agents. Our method not only paves the path towards autotelic LLM agents that learn online but can also outperform on-policy methods in more classic multi-goal RL environments.