Statistical Inference under Adaptive Sampling with LinUCB

TL;DR

This paper proves that the LinUCB algorithm in linear bandits satisfies stability, leading to asymptotic normality of estimators and enabling the construction of tighter confidence sets and hypothesis tests in adaptive sampling.

Contribution

It establishes a central limit theorem for LinUCB, characterizing the eigenstructure of the covariance matrix and deriving asymptotically valid confidence sets.

Findings

LinUCB satisfies the stability property.

Eigenvalues and eigenvectors of the covariance matrix are characterized.

Asymptotic normality of the estimation error is proven.

Abstract

Adaptively collected data has become ubiquitous within modern practice. However, even seemingly benign adaptive sampling schemes can introduce severe biases, rendering traditional statistical inference tools inapplicable. This can be mitigated by a property called stability, which states that if the rate at which an algorithm takes actions converges to a deterministic limit, one can expect that certain parameters are asymptotically normal. Building on a recent line of work for the multi-armed bandit setting, we show that the linear upper confidence bound (LinUCB) algorithm for linear bandits satisfies this property. In doing so, we painstakingly characterize the behavior of the eigenvalues and eigenvectors of the random design feature covariance matrix in the setting where the action set is the unit ball, showing that it decomposes into a rank-one direction that locks onto the true parameter and an almost-isotropic bulk that grows at a predictable $\sqrt{T}$ rate. This allows us to establish a central limit theorem for the LinUCB algorithm, establishing asymptotic normality for the limiting distribution of the estimation error where the convergence occurs at a $T^{-1/4}$ rate. The resulting Wald-type confidence sets and hypothesis tests do not depend on the feature covariance matrix and are asymptotically tighter than existing nonasymptotic confidence sets. Numerical simulations corroborate our findings.