A Reduction Algorithm for Markovian Contextual Linear Bandits

TL;DR

This paper extends the reduction approach for linear contextual bandits to Markovian contexts, enabling efficient algorithms with strong regret guarantees even with temporally correlated data.

Contribution

It introduces a reduction method for Markovian contextual linear bandits under geometric ergodicity, including an online learning algorithm for unknown transition distributions.

Findings

Achieves regret bounds comparable to standard linear bandit algorithms.

Handles temporally correlated contexts via a stationary surrogate action set.

Provides a phased algorithm with online transition learning.

Abstract

Recent work shows that when contexts are drawn i.i.d., linear contextual bandits can be reduced to single-context linear bandits. This ``contexts are cheap" perspective is highly advantageous, as it allows for sharper finite-time analyses and leverages mature techniques from the linear bandit literature, such as those for misspecification and adversarial corruption. Motivated by applications with temporally correlated availability, we extend this perspective to Markovian contextual linear bandits, where the action set evolves via an exogenous Markov chain. Our main contribution is a reduction that applies under uniform geometric ergodicity. We construct a stationary surrogate action set to solve the problem using a standard linear bandit oracle, employing a delayed-update scheme to control the bias induced by the nonstationary conditional context distributions. We further provide a phased algorithm for unknown transition distributions that learns the surrogate mapping online. In both settings, we obtain a high-probability worst-case regret bound matching that of the underlying linear bandit oracle, with only lower-order dependence on the mixing time.