From Words to Actions: Unveiling the Theoretical Underpinnings of LLM-Driven Autonomous Systems

TL;DR

This paper provides a theoretical understanding of how large language models enable autonomous decision-making in physical environments through hierarchical reinforcement learning, Bayesian imitation, and exploration strategies.

Contribution

It introduces a theoretical framework linking LLMs to Bayesian imitation learning and explores exploration strategies, extending to multi-agent coordination.

Findings

LLM planners perform Bayesian aggregated imitation learning under certain assumptions.

Naive execution of LLM-derived subgoals results in linear regret.

An epsilon-greedy exploration strategy achieves sublinear regret with small pretraining error.

Abstract

In this work, from a theoretical lens, we aim to understand why large language model (LLM) empowered agents are able to solve decision-making problems in the physical world. To this end, consider a hierarchical reinforcement learning (RL) model where the LLM Planner and the Actor perform high-level task planning and low-level execution, respectively. Under this model, the LLM Planner navigates a partially observable Markov decision process (POMDP) by iteratively generating language-based subgoals via prompting. Under proper assumptions on the pretraining data, we prove that the pretrained LLM Planner effectively performs Bayesian aggregated imitation learning (BAIL) through in-context learning. Additionally, we highlight the necessity for exploration beyond the subgoals derived from BAIL by proving that naively executing the subgoals returned by LLM leads to a linear regret. As a remedy, we introduce an $\epsilon$-greedy exploration strategy to BAIL, which is proven to incur sublinear regret when the pretraining error is small. Finally, we extend our theoretical framework to include scenarios where the LLM Planner serves as a world model for inferring the transition model of the environment and to multi-agent settings, enabling coordination among multiple Actors.