Offline Learning for Combinatorial Multi-armed Bandits

TL;DR

This paper introduces Off-CMAB, a novel offline learning framework for combinatorial multi-armed bandits, addressing the costs of online interaction and leveraging offline data with theoretical guarantees and practical applications.

Contribution

It proposes the CLCB algorithm and data coverage conditions, providing the first offline learning approach for CMAB with near-optimal theoretical guarantees.

Findings

CLCB achieves near-optimal suboptimality gap under data coverage conditions.

Off-CMAB effectively handles nonlinear rewards and out-of-distribution actions.

Experimental results demonstrate superior performance on synthetic and real datasets.

Abstract

The combinatorial multi-armed bandit (CMAB) is a fundamental sequential decision-making framework, extensively studied over the past decade. However, existing work primarily focuses on the online setting, overlooking the substantial costs of online interactions and the readily available offline datasets. To overcome these limitations, we introduce Off-CMAB, the first offline learning framework for CMAB. Central to our framework is the combinatorial lower confidence bound (CLCB) algorithm, which combines pessimistic reward estimations with combinatorial solvers. To characterize the quality of offline datasets, we propose two novel data coverage conditions and prove that, under these conditions, CLCB achieves a near-optimal suboptimality gap, matching the theoretical lower bound up to a logarithmic factor. We validate Off-CMAB through practical applications, including learning to rank, large language model (LLM) caching, and social influence maximization, showing its ability to handle nonlinear reward functions, general feedback models, and out-of-distribution action samples that excludes optimal or even feasible actions. Extensive experiments on synthetic and real-world datasets further highlight the superior performance of CLCB.