Leveraging Offline Data in Linear Latent Contextual Bandits

TL;DR

This paper introduces algorithms for linear latent contextual bandits that effectively leverage offline data to improve online decision-making, with theoretical guarantees and practical validation on real datasets.

Contribution

It proposes the first end-to-end algorithms for linear latent bandits that handle uncountably many latent states, including an offline subspace learning method and two online algorithms with optimal regret bounds.

Findings

Offline subspace learning with provable guarantees

Online algorithms with minimax optimal regret bounds

Validated effectiveness on synthetic and real-world data

Abstract

Leveraging offline data is an attractive way to accelerate online sequential decision-making. However, it is crucial to account for latent states in users or environments in the offline data, and latent bandits form a compelling model for doing so. In this light, we design end-to-end latent bandit algorithms capable of handing uncountably many latent states. We focus on a linear latent contextual bandit $-$ a linear bandit where each user has its own high-dimensional reward parameter in $\mathbb{R}^{d_A}$, but reward parameters across users lie in a low-rank latent subspace of dimension $d_K \ll d_A$. First, we provide an offline algorithm to learn this subspace with provable guarantees. We then present two online algorithms that utilize the output of this offline algorithm to accelerate online learning. The first enjoys $\tilde{O}(\min(d_A\sqrt{T}, d_K\sqrt{T}(1+\sqrt{d_AT/d_KN})))$ regret guarantees, so that the effective dimension is lower when the size $N$ of the offline dataset is larger. We prove a matching lower bound on regret, showing that our algorithm is minimax optimal. The second is a practical algorithm that enjoys only a slightly weaker guarantee, but is computationally efficient. We also establish the efficacy of our methods using experiments on both synthetic data and real-life movie recommendation data from MovieLens. Finally, we theoretically establish the generality of the latent bandit model by proving a de Finetti theorem for stateless decision processes.