On Submodular Contextual Bandits

TL;DR

This paper introduces algorithms for contextual bandit problems with submodular reward functions and matroid constraints, achieving regret bounds that depend on an online regression oracle and improving understanding of adaptive decision-making in complex combinatorial settings.

Contribution

It proposes a novel algorithmic approach for submodular contextual bandits with matroid constraints, providing regret bounds based on an online regression oracle and local randomization techniques.

Findings

Regret scales as O(√(n * Reg(𝓕))) with the proposed algorithm.

An ε-Greedy method attains regret of O(n^{2/3} * Reg(𝓕)^{1/3}) against a stronger benchmark.

The algorithms effectively handle time-varying constraints and unknown submodular reward functions.

Abstract

We consider the problem of contextual bandits where actions are subsets of a ground set and mean rewards are modeled by an unknown monotone submodular function that belongs to a class $\mathcal{F}$. We allow time-varying matroid constraints to be placed on the feasible sets. Assuming access to an online regression oracle with regret $\mathsf{Reg}(\mathcal{F})$, our algorithm efficiently randomizes around local optima of estimated functions according to the Inverse Gap Weighting strategy. We show that cumulative regret of this procedure with time horizon $n$ scales as $O(\sqrt{n \mathsf{Reg}(\mathcal{F})})$ against a benchmark with a multiplicative factor $1/2$. On the other hand, using the techniques of (Filmus and Ward 2014), we show that an $\epsilon$-Greedy procedure with local randomization attains regret of $O(n^{2/3} \mathsf{Reg}(\mathcal{F})^{1/3})$ against a stronger $(1-e^{-1})$ benchmark.