Retrospex: Language Agent Meets Offline Reinforcement Learning Critic

TL;DR

Retrospex is a novel LLM agent framework that leverages offline reinforcement learning to analyze past experiences, improving decision-making without directly embedding experiences into the language model's context.

Contribution

It introduces a new framework combining LLMs with an offline RL critic and dynamic rescoring, enhancing agent performance by better utilizing past experiences.

Findings

Retrospex outperforms strong baselines in ScienceWorld, ALFWorld, and Webshop environments.

The offline RL critic effectively guides LLM actions based on past experiences.

Dynamic rescoring improves task-specific decision-making.

Abstract

Large Language Models (LLMs) possess extensive knowledge and commonsense reasoning capabilities, making them valuable for creating powerful agents. However, existing LLM agent frameworks have not fully utilized past experiences for improvement. This work introduces a new LLM-based agent framework called Retrospex, which addresses this challenge by analyzing past experiences in depth. Unlike previous approaches, Retrospex does not directly integrate experiences into the LLM's context. Instead, it combines the LLM's action likelihood with action values estimated by a Reinforcement Learning (RL) Critic, which is trained on past experiences through an offline ''retrospection'' process. Additionally, Retrospex employs a dynamic action rescoring mechanism that increases the importance of experience-based values for tasks that require more interaction with the environment. We evaluate Retrospex in ScienceWorld, ALFWorld and Webshop environments, demonstrating its advantages over strong, contemporary baselines.