Efficient Sequential Decision Making with Large Language Models

TL;DR

This paper introduces an efficient method for sequential decision making using large language models by leveraging online model selection algorithms, significantly reducing computational costs while improving performance.

Contribution

The paper proposes a novel approach that combines online model selection with LLMs, avoiding costly retraining and achieving superior decision-making performance.

Findings

Outperforms traditional decision algorithms and vanilla LLM agents.

Achieves over 6x performance gain on Amazon dataset.

Requires only 1.5% of LLM calls compared to baselines.

Abstract

This paper focuses on extending the success of large language models (LLMs) to sequential decision making. Existing efforts either (i) re-train or finetune LLMs for decision making, or (ii) design prompts for pretrained LLMs. The former approach suffers from the computational burden of gradient updates, and the latter approach does not show promising results. In this paper, we propose a new approach that leverages online model selection algorithms to efficiently incorporate LLMs agents into sequential decision making. Statistically, our approach significantly outperforms both traditional decision making algorithms and vanilla LLM agents. Computationally, our approach avoids the need for expensive gradient updates of LLMs, and throughout the decision making process, it requires only a small number of LLM calls. We conduct extensive experiments to verify the effectiveness of our proposed approach. As an example, on a large-scale Amazon dataset, our approach achieves more than a 6x performance gain over baselines while calling LLMs in only 1.5% of the time steps.