Matrix games with bandit feedback

TL;DR

This paper investigates zero-sum matrix games with unknown payoffs and bandit feedback, proposing new algorithms with regret guarantees that outperform existing methods, and highlighting the failure of Thompson sampling in this context.

Contribution

It introduces regret analyses for variants of UCB and K-learning tailored for matrix games with bandit feedback, applicable against any opponent, including adversarial ones.

Findings

UCB and K-learning variants achieve low regret in matrix games.

Thompson sampling performs poorly in this setting.

New algorithms outperform existing methods empirically.

Abstract

We study a version of the classical zero-sum matrix game with unknown payoff matrix and bandit feedback, where the players only observe each others actions and a noisy payoff. This generalizes the usual matrix game, where the payoff matrix is known to the players. Despite numerous applications, this problem has received relatively little attention. Although adversarial bandit algorithms achieve low regret, they do not exploit the matrix structure and perform poorly relative to the new algorithms. The main contributions are regret analyses of variants of UCB and K-learning that hold for any opponent, e.g., even when the opponent adversarially plays the best-response to the learner's mixed strategy. Along the way, we show that Thompson fails catastrophically in this setting and provide empirical comparison to existing algorithms.