Pessimism for Offline Linear Contextual Bandits using $\ell_p$ Confidence Sets

TL;DR

This paper introduces a family of pessimistic offline learning algorithms for linear contextual bandits based on $\, ext{ell}_p$ confidence sets, highlighting a new $\, ext{ell}_\, ext{infty}$ variant that is adaptively optimal.

Contribution

It proposes a novel $\, ext{ell}_\, ext{infty}$ confidence set-based learning rule that outperforms existing methods in linear contextual bandit offline learning.

Findings

The $\, ext{ell}_\, ext{infty}$ rule is adaptively minimax optimal.

The $\, ext{ell}_\, ext{infty}$ rule dominates other predictors in the family.

The approach generalizes lower confidence bounds to the linear setting.

Abstract

We present a family $\{\hat{\pi}\}_{p\ge 1}$ of pessimistic learning rules for offline learning of linear contextual bandits, relying on confidence sets with respect to different $\ell_p$ norms, where $\hat{\pi}_2$ corresponds to Bellman-consistent pessimism (BCP), while $\hat{\pi}_\infty$ is a novel generalization of lower confidence bound (LCB) to the linear setting. We show that the novel $\hat{\pi}_\infty$ learning rule is, in a sense, adaptively optimal, as it achieves the minimax performance (up to log factors) against all $\ell_q$-constrained problems, and as such it strictly dominates all other predictors in the family, including $\hat{\pi}_2$.