Enhancing Decision-Making of Large Language Models via Actor-Critic

TL;DR

This paper introduces a novel Actor-Critic framework for large language models that enhances their decision-making in complex, multi-step environments by improving long-term policy evaluation and optimization.

Contribution

The paper presents the LAC framework, a scalable, gradient-free method that improves LLM decision-making by integrating Q-value estimation and long-term reasoning.

Findings

LAC outperforms state-of-the-art methods across diverse environments.

Achieves competitive results with 7B/8B parameter LLMs, surpassing GPT-4 baselines.

Demonstrates generality and effectiveness in high-level and large action space tasks.

Abstract

Large Language Models (LLMs) have achieved remarkable advancements in natural language processing tasks, yet they encounter challenges in complex decision-making scenarios that require long-term reasoning and alignment with high-level objectives. Existing methods either rely on short-term auto-regressive action generation or face limitations in accurately simulating rollouts and assessing outcomes, leading to sub-optimal decisions. This paper introduces a novel LLM-based Actor-Critic framework, termed LAC, that effectively improves LLM policies with long-term action evaluations in a principled and scalable way. Our approach addresses two key challenges: (1) extracting robust action evaluations by computing Q-values via token logits associated with positive/negative outcomes, enhanced by future trajectory rollouts and reasoning; and (2) enabling efficient policy improvement through a gradient-free mechanism. Experiments across diverse environments -- including high-level decision-making (ALFWorld), low-level action spaces (BabyAI-Text), and large action spaces (WebShop) -- demonstrate the framework's generality and superiority over state-of-the-art methods. Notably, our approach achieves competitive performance using 7B/8B parameter LLMs, even outperforming baseline methods employing GPT-4 in complex tasks. These results underscore the potential of integrating structured policy optimization with LLMs' intrinsic knowledge to advance decision-making capabilities in multi-step environments.