Latent Order Bandits

TL;DR

Latent order bandits (LOB) improve sequential decision-making by relaxing prior assumptions, using partial order knowledge to adapt to varying reward distributions across instances, with proven regret bounds and competitive empirical results.

Contribution

LOB introduces a new approach that requires only partial order knowledge, broadening applicability over traditional latent bandits with full posterior models.

Findings

LOB algorithms are competitive with full-prior latent bandits when reward parameters are shared.

LOB outperforms full-prior models when reward scales differ across instances.

The paper provides regret bounds and demonstrates empirical effectiveness of LOB methods.

Abstract

Bandit algorithms solve diverse sequential decision-making problems, but are often too sample-inefficient for from-scratch personalization. To substantially reduce exploration times, latent bandit algorithms exploit cross-instance structure implied by discrete latent states, provided that the posterior distribution of rewards and latent states is known and accurate. However, obtaining an accurate model of this structure is difficult, and a small number of latent states may be insufficient to characterize the reward distributions in all problem instances. We propose latent order bandits (LOB), relaxing the assumptions of latent bandits to require only prior knowledge of a partial order of action preferences in each state. This allows instances of the same state to vary in reward distributions, as long as the partial order of actions is shared. For example, groups of users on a streaming service may agree on which movie genres are the best but rate experiences on different scales. We give an upper-confidence bound procedure for the LOB problem, applicable to both total and partial latent orders, and give an upper bound on its regret. To improve empirical performance, we propose a posterior-sampling algorithm and show, in a suite of experiments, that both are competitive with full-prior latent bandits when same-state instances share reward parameters, and preferable to them when reward scales differ between instances with the same latent state.