Phased Exploration with Greedy Exploitation in Stochastic Combinatorial Partial Monitoring Games

TL;DR

This paper introduces a new phased exploration algorithm for combinatorial partial monitoring games that improves regret bounds and simplifies implementation compared to previous methods, especially in large action spaces.

Contribution

It proposes a novel PEGE framework that reduces complexity and relaxes assumptions, and introduces PEGE2 achieving optimal regret without large action space dependence.

Findings

PEGE achieves $O(T^{2/3}\sqrt{\log T})$ distribution independent regret.

PEGE2 attains $O(\log T)$ regret, matching previous best but with less dependence on action space size.

The algorithms are applicable to practical online ranking problems with partial feedback.

Abstract

Partial monitoring games are repeated games where the learner receives feedback that might be different from adversary's move or even the reward gained by the learner. Recently, a general model of combinatorial partial monitoring (CPM) games was proposed \cite{lincombinatorial2014}, where the learner's action space can be exponentially large and adversary samples its moves from a bounded, continuous space, according to a fixed distribution. The paper gave a confidence bound based algorithm (GCB) that achieves $O(T^{2/3}\log T)$ distribution independent and $O(\log T)$ distribution dependent regret bounds. The implementation of their algorithm depends on two separate offline oracles and the distribution dependent regret additionally requires existence of a unique optimal action for the learner. Adopting their CPM model, our first contribution is a Phased Exploration with Greedy Exploitation (PEGE) algorithmic framework for the problem. Different algorithms within the framework achieve $O(T^{2/3}\sqrt{\log T})$ distribution independent and $O(\log^2 T)$ distribution dependent regret respectively. Crucially, our framework needs only the simpler "argmax" oracle from GCB and the distribution dependent regret does not require existence of a unique optimal action. Our second contribution is another algorithm, PEGE2, which combines gap estimation with a PEGE algorithm, to achieve an $O(\log T)$ regret bound, matching the GCB guarantee but removing the dependence on size of the learner's action space. However, like GCB, PEGE2 requires access to both offline oracles and the existence of a unique optimal action. Finally, we discuss how our algorithm can be efficiently applied to a CPM problem of practical interest: namely, online ranking with feedback at the top.