Large Language Models for Sequential Decision-Making: Improving In-Context Learning via Supervised Fine-Tuning

TL;DR

This paper demonstrates that supervised fine-tuning of large language models significantly enhances their ability to perform sequential decision-making tasks from offline data, outperforming in-context learning alone.

Contribution

The paper introduces a framework for fine-tuning LLMs for sequential decision-making, providing theoretical insights and empirical evidence of improved performance in complex environments.

Findings

Fine-tuned LLMs achieve smaller optimality gaps than in-context-only baselines.

Supervised fine-tuning improves decision-making in longer-horizon, partially observed, and ambiguous environments.

Theoretical analysis links attention mechanisms to optimal Q-function estimation.

Abstract

Large language models (LLMs) have shown remarkable in-context learning (ICL) capabilities, yet their potential for sequential decision-making remains underexplored. In this paper, we study the ICL capabilities of LLMs in sequential decision-making settings, including Markov Decision Processes (MDPs), Partially Observable MDPs (POMDPs), and Ambiguous POMDPs (APOMDPs). We fine-tune pretrained LLMs to perform few-shot decision-making directly from offline, oracle-labeled trajectories. Our framework enables flexible imitation of policies through supervised fine-tuning (SFT). Theoretically, we focus on linear MDPs and interpret a fine-tuned attention layer as implicitly estimating optimal Q-functions from in-context data. Building on this interpretation, we derive an end-to-end suboptimality bound for the induced policy that separates the in-context estimation error from the training-length bias. Empirically, across synthetic MDP, POMDP, and APOMDP settings, we find that fine-tuned LLMs achieve substantially smaller optimality gaps than in-context-only and random baselines, with especially large gains in longer-horizon, partially observed, and model-ambiguous environments. Together, these results show that supervised fine-tuning provides an effective route to endowing pretrained LLMs with sequential decision-making capabilities from offline data, which is an important advantage in domains such as healthcare where offline data are abundant.