Online Bandit Learning against an Adaptive Adversary: from Regret to Policy Regret

TL;DR

This paper introduces the concept of policy regret for online learning against adaptive adversaries, showing limitations and proposing a transformation technique to achieve sublinear policy regret when adversaries have bounded memory.

Contribution

It defines policy regret as a more meaningful measure against adaptive adversaries and provides a method to convert existing regret guarantees into policy regret guarantees under bounded memory adversaries.

Findings

No bandit algorithm can guarantee sublinear policy regret against unbounded memory adaptive adversaries.

A general technique converts any sublinear regret bandit algorithm into one with sublinear policy regret against bounded memory adversaries.

The results extend to switching, internal, and swap regret variants.

Abstract

Online learning algorithms are designed to learn even when their input is generated by an adversary. The widely-accepted formal definition of an online algorithm's ability to learn is the game-theoretic notion of regret. We argue that the standard definition of regret becomes inadequate if the adversary is allowed to adapt to the online algorithm's actions. We define the alternative notion of policy regret, which attempts to provide a more meaningful way to measure an online algorithm's performance against adaptive adversaries. Focusing on the online bandit setting, we show that no bandit algorithm can guarantee a sublinear policy regret against an adaptive adversary with unbounded memory. On the other hand, if the adversary's memory is bounded, we present a general technique that converts any bandit algorithm with a sublinear regret bound into an algorithm with a sublinear policy regret bound. We extend this result to other variants of regret, such as switching regret, internal regret, and swap regret.