A Framework for Adapting Offline Algorithms to Solve Combinatorial Multi-Armed Bandit Problems with Bandit Feedback

TL;DR

This paper introduces a versatile framework that transforms offline algorithms into online solutions for complex combinatorial bandit problems, achieving low regret with minimal feedback and broad applicability.

Contribution

The paper presents a novel framework that adapts offline approximation algorithms into bandit algorithms with sublinear regret, applicable to non-linear reward functions and requiring only black-box access.

Findings

Achieves $ ilde{O}(T^{2/3})$ regret dependence on horizon T.

Framework is robust to small errors in function evaluation.

Outperforms existing methods in submodular maximization with real-world data.

Abstract

We investigate the problem of stochastic, combinatorial multi-armed bandits where the learner only has access to bandit feedback and the reward function can be non-linear. We provide a general framework for adapting discrete offline approximation algorithms into sublinear $\alpha$-regret methods that only require bandit feedback, achieving $\mathcal{O}\left(T^\frac{2}{3}\log(T)^\frac{1}{3}\right)$ expected cumulative $\alpha$-regret dependence on the horizon $T$. The framework only requires the offline algorithms to be robust to small errors in function evaluation. The adaptation procedure does not even require explicit knowledge of the offline approximation algorithm -- the offline algorithm can be used as a black box subroutine. To demonstrate the utility of the proposed framework, the proposed framework is applied to diverse applications in submodular maximization. The new CMAB algorithms for submodular maximization with knapsack constraints outperform a full-bandit method developed for the adversarial setting in experiments with real-world data.