Nonstochastic Bandits with Infinitely Many Experts

TL;DR

This paper extends nonstochastic bandit algorithms to handle infinitely many experts, providing regret bounds and a meta-algorithm that adapts to the best expert position with high probability.

Contribution

It introduces a variant of Exp4.P for finite experts, extends it to infinitely many experts, and provides regret bounds and analysis for structured experts.

Findings

Regret bound of  ig( i^*K + \u007f ig) for the meta-algorithm.

The algorithm achieves near-optimal regret when the best expert is not too deep.

In structured settings, learning can be expedited, improving practical performance.

Abstract

We study the problem of nonstochastic bandits with expert advice, extending the setting from finitely many experts to any countably infinite set: A learner aims to maximize the total reward by taking actions sequentially based on bandit feedback while benchmarking against a set of experts. We propose a variant of Exp4.P that, for finitely many experts, enables inference of correct expert rankings while preserving the order of the regret upper bound. We then incorporate the variant into a meta-algorithm that works on infinitely many experts. We prove a high-probability upper bound of $\tilde{\mathcal{O}} \big( i^*K + \sqrt{KT} \big)$ on the regret, up to polylog factors, where $i^*$ is the unknown position of the best expert, $K$ is the number of actions, and $T$ is the time horizon. We also provide an example of structured experts and discuss how to expedite learning in such case. Our meta-learning algorithm achieves optimal regret up to polylog factors when $i^* = \tilde{\mathcal{O}} \big( \sqrt{T/K} \big)$. If a prior distribution is assumed to exist for $i^*$, the probability of optimality increases with $T$, the rate of which can be fast.