Stochastic Linear Bandits with Protected Subspace

TL;DR

This paper introduces a stochastic linear bandit model with a protected subspace, proposing an OFUL-based algorithm that estimates the subspace and achieves near-optimal regret bounds, while highlighting limitations of optimism in certain settings.

Contribution

It develops a novel bandit framework with protected subspace constraints, providing regret bounds and empirical validation, and demonstrating the limitations of optimism-based algorithms.

Findings

Achieves $ ilde{O}(sdrac{ ext{sqrt}(T)}{})$ regret in continuous action spaces.

Shows linear regret can occur with discrete actions under optimism.

Establishes a lower bound of $ ext{Omega}(T^{3/4})$ for certain protected subspace settings.

Abstract

We study a variant of the stochastic linear bandit problem wherein we optimize a linear objective function but rewards are accrued only orthogonal to an unknown subspace (which we interpret as a \textit{protected space}) given only zero-order stochastic oracle access to both the objective itself and protected subspace. In particular, at each round, the learner must choose whether to query the objective or the protected subspace alongside choosing an action. Our algorithm, derived from the OFUL principle, uses some of the queries to get an estimate of the protected space, and (in almost all rounds) plays optimistically with respect to a confidence set for this space. We provide a $\tilde{O}(sd\sqrt{T})$ regret upper bound in the case where the action space is the complete unit ball in $\mathbb{R}^d$, $s < d$ is the dimension of the protected subspace, and $T$ is the time horizon. Moreover, we demonstrate that a discrete action space can lead to linear regret with an optimistic algorithm, reinforcing the sub-optimality of optimism in certain settings. We also show that protection constraints imply that for certain settings, no consistent algorithm can have a regret smaller than $\Omega(T^{3/4}).$ We finally empirically validate our results with synthetic and real datasets.