SAND: Boosting LLM Agents with Self-Taught Action Deliberation

TL;DR

This paper introduces SAND, a framework that enhances LLM agents by enabling explicit action deliberation and self-critique, leading to significant performance improvements over traditional finetuning methods.

Contribution

The paper presents a novel self-taught deliberation framework for LLM agents, improving action exploration and decision quality through iterative self-critique and finetuning.

Findings

SAND achieves 20% average improvement over initial finetuning.

Outperforms state-of-the-art agent tuning approaches.

Effective in two interactive agent tasks.

Abstract

Large Language Model (LLM) agents are commonly tuned with supervised finetuning on ReAct-style expert trajectories or preference optimization over pairwise rollouts. Most of these methods focus on imitating specific expert behaviors or promoting chosen reasoning thoughts and actions over rejected ones. However, without reasoning and comparing over alternatives actions, LLM agents finetuned with these methods may over-commit towards seemingly plausible but suboptimal actions due to limited action space exploration. To address this, in this paper we propose Self-taught ActioN Deliberation (SAND) framework, enabling LLM agents to explicitly deliberate over candidate actions before committing to one. To tackle the challenges of when and what to deliberate given large action space and step-level action evaluation, we incorporate self-consistency action sampling and execution-guided action critique to help synthesize step-wise action deliberation thoughts using the base model of the LLM agent. In an iterative manner, the deliberation trajectories are then used to finetune the LLM agent itself. Evaluating on two representative interactive agent tasks, SAND achieves an average 20% improvement over initial supervised finetuning and also outperforms state-of-the-art agent tuning approaches.