Small-loss bounds for online learning with partial information

TL;DR

This paper introduces algorithms for adversarial online learning with partial feedback that achieve data-dependent small-loss regret bounds, extending to various settings like semi-bandits and contextual bandits, with optimal guarantees.

Contribution

It provides the first data-dependent small-loss regret bounds for general feedback graphs and extends these results to multiple online learning scenarios using a black-box approach.

Findings

Achieved small-loss regret bounds of o(α L*) with high probability.

Extended results to semi-bandits, contextual bandits, and shifting comparators.

Provided optimal bounds for classical bandit and semi-bandit problems, answering open questions.

Abstract

We consider the problem of adversarial (non-stochastic) online learning with partial information feedback, where at each round, a decision maker selects an action from a finite set of alternatives. We develop a black-box approach for such problems where the learner observes as feedback only losses of a subset of the actions that includes the selected action. When losses of actions are non-negative, under the graph-based feedback model introduced by Mannor and Shamir, we offer algorithms that attain the so called "small-loss" $o(\alpha L^{\star})$ regret bounds with high probability, where $\alpha$ is the independence number of the graph, and $L^{\star}$ is the loss of the best action. Prior to our work, there was no data-dependent guarantee for general feedback graphs even for pseudo-regret (without dependence on the number of actions, i.e. utilizing the increased information feedback). Taking advantage of the black-box nature of our technique, we extend our results to many other applications such as semi-bandits (including routing in networks), contextual bandits (even with an infinite comparator class), as well as learning with slowly changing (shifting) comparators. In the special case of classical bandit and semi-bandit problems, we provide optimal small-loss, high-probability guarantees of $\tilde{O}(\sqrt{dL^{\star}})$ for actual regret, where $d$ is the number of actions, answering open questions of Neu. Previous bounds for bandits and semi-bandits were known only for pseudo-regret and only in expectation. We also offer an optimal $\tilde{O}(\sqrt{\kappa L^{\star}})$ regret guarantee for fixed feedback graphs with clique-partition number at most $\kappa$.