Bandits with Stochastic Experts: Constant Regret, Empirical Experts and Episodes

TL;DR

This paper introduces algorithms for a variant of the contextual bandit problem involving stochastic experts, achieving constant regret bounds and effective performance in episodic environments with changing contexts and rewards.

Contribution

The paper proposes the D-UCB and ED-UCB algorithms that leverage importance sampling for expert sharing and handle approximate knowledge, with proven regret bounds in both fixed and episodic settings.

Findings

D-UCB achieves horizon-independent constant regret.

ED-UCB performs well with approximate expert knowledge.

Regret scales as O(E(N+1) + N√E/T²) over episodes.

Abstract

We study a variant of the contextual bandit problem where an agent can intervene through a set of stochastic expert policies. Given a fixed context, each expert samples actions from a fixed conditional distribution. The agent seeks to remain competitive with the 'best' among the given set of experts. We propose the Divergence-based Upper Confidence Bound (D-UCB) algorithm that uses importance sampling to share information across experts and provide horizon-independent constant regret bounds that only scale linearly in the number of experts. We also provide the Empirical D-UCB (ED-UCB) algorithm that can function with only approximate knowledge of expert distributions. Further, we investigate the episodic setting where the agent interacts with an environment that changes over episodes. Each episode can have different context and reward distributions resulting in the best expert changing across episodes. We show that by bootstrapping from $\mathcal{O}\left(N\log\left(NT^2\sqrt{E}\right)\right)$ samples, ED-UCB guarantees a regret that scales as $\mathcal{O}\left(E(N+1) + \frac{N\sqrt{E}}{T^2}\right)$ for $N$ experts over $E$ episodes, each of length $T$. We finally empirically validate our findings through simulations.