Beyond Numeric Rewards: In-Context Dueling Bandits with LLM Agents

TL;DR

This paper explores the zero-shot decision-making capabilities of Large Language Models in dueling bandits, introduces an agentic framework called LEAD to improve performance, and provides theoretical and empirical validation of its effectiveness.

Contribution

It demonstrates LLMs' potential in cross-domain in-context reinforcement learning for dueling bandits and proposes LEAD, a novel framework integrating classic algorithms with LLMs for improved regret performance.

Findings

LLMs exhibit low short-term regret in dueling bandits without training.

LEAD framework inherits theoretical guarantees from classic algorithms.

LEAD is robust to noisy and adversarial prompts.

Abstract

In-Context Reinforcement Learning (ICRL) is a frontier paradigm to solve Reinforcement Learning (RL) problems in the foundation model era. While ICRL capabilities have been demonstrated in transformers through task-specific training, the potential of Large Language Models (LLMs) out-of-the-box remains largely unexplored. This paper investigates whether LLMs can generalize cross-domain to perform ICRL under the problem of Dueling Bandits (DB), a stateless preference-based RL setting. We find that the top-performing LLMs exhibit a notable zero-shot capacity for relative decision-making, which translates to low short-term weak regret across all DB environment instances by quickly including the best arm in duels. However, an optimality gap still exists between LLMs and classic DB algorithms in terms of strong regret. LLMs struggle to converge and consistently exploit even when explicitly prompted to do so, and are sensitive to prompt variations. To bridge this gap, we propose an agentic flow framework: LLM with Enhanced Algorithmic Dueling (LEAD), which integrates off-the-shelf DB algorithm support with LLM agents through fine-grained adaptive interplay. We show that LEAD has theoretical guarantees inherited from classic DB algorithms on both weak and strong regret. We validate its efficacy and robustness even with noisy and adversarial prompts. The design of such an agentic framework sheds light on how to enhance the trustworthiness of general-purpose LLMs generalized to in-context decision-making tasks.