No-Regret Linear Bandits beyond Realizability

TL;DR

This paper introduces a new model of misspecification in linear bandits that depends on the suboptimality gap, and shows that the classical LinUCB algorithm remains robust under this model, achieving near-optimal regret.

Contribution

It proposes a gap-dependent misspecification model for linear bandits and demonstrates that LinUCB is robust and effective under this new framework.

Findings

LinUCB achieves near-optimal $\

The new model captures realistic misspecification scenarios where errors are proportional to suboptimality gaps.

A novel self-bounding proof technique is developed to analyze regret under misspecification.

Abstract

We study linear bandits when the underlying reward function is not linear. Existing work relies on a uniform misspecification parameter $\epsilon$ that measures the sup-norm error of the best linear approximation. This results in an unavoidable linear regret whenever $\epsilon > 0$. We describe a more natural model of misspecification which only requires the approximation error at each input $x$ to be proportional to the suboptimality gap at $x$. It captures the intuition that, for optimization problems, near-optimal regions should matter more and we can tolerate larger approximation errors in suboptimal regions. Quite surprisingly, we show that the classical LinUCB algorithm -- designed for the realizable case -- is automatically robust against such gap-adjusted misspecification. It achieves a near-optimal $\sqrt{T}$ regret for problems that the best-known regret is almost linear in time horizon $T$. Technically, our proof relies on a novel self-bounding argument that bounds the part of the regret due to misspecification by the regret itself.

Equations

$$R_T(x_0,...,x_{T-1})=\sum _{t=0}^{T-1}{r}_t=\sum _{t=0}^{T-1}\underset{x\in \mathcal{X}}{\max }{f}_0(x)-f_0(x_t),$$

$$x_{\ast }={\mathrm{argmax}}_{x\in \mathcal{X}}{f}_0(x),$$

$$y_t=f_0(x_t)+{\eta }_t,$$

$$\underset{x\in \mathcal{X}}{\sup }\left|\frac{f(x)-f_0(x)}{f^{\ast }-f_0(x)}\right|\le \rho .$$

$${\mathrm{argmax}}_{x}f(x)={\mathrm{argmax}}_x{f}_0(x).$$

$$\underset{x\in \mathcal{X}}{\max }f(x)=f^{\ast }.$$

$$|f(x)-f_0(x)|\le \rho (f^{\ast }-f_0(x))=\rho r(x).$$

$$\underset{x\in \mathcal{X}}{\sup }\left|\frac{f(x)-f_w^{\ast }+f^{\ast }-f_0(x)}{f^{\ast }-f_0(x)}\right|\le \rho .$$

$${\mathrm{argmax}}_{x\in \mathcal{X}}{f}_0(x)\subset \{x\in {\mathbb{R}}^d:x^{\top }a=b\}.$$

$$\rho <\frac{1}{8d\sqrt{\log \left(1+\frac{T{C}_b^2{C}_w^2}{d{\sigma }^2}\right)}}=O\left(\frac{1}{d\sqrt{\log T}}\right).$$

$${\Sigma }_{t+1}=\lambda I+\sum _{i=0}^t{x}_i{x}_i^{\top }\mathrm{where}{\Sigma }_0=\lambda I.$$

$${\hat{w}}_{t+1}={\mathrm{argmin}}_w\lambda \Vert w{\Vert }_2^2+\sum _{i=0}^t(w^{\top }{x}_i-y_i{)}_2^2.$$

$${\beta }_t=8{\sigma }^2\left(1+d\log \left(1+\frac{t{C}_b^2{C}_w^2}{d{\sigma }^2}\right)+2\log \left(\frac{{\pi }^2{t}^2}{3\delta }\right)\right).$$

$$R_T$$

$$\le F+\sqrt{\left(\sum _{t=1}^{T-1}1\right)\left(\sum _{t=1}^{T-1}{r}_t^2\right)}$$

$$=F+\sqrt{(T-1)\sum _{t=1}^{T-1}{r}_t^2}.$$

$${\Delta }_t=f_0(x_t)-w_{\ast }^{\top }{x}_t,$$

$$y_t=f_0(x_t)+{\eta }_t=w_{\ast }^{\top }{x}_t+{\Delta }_t+{\eta }_t.$$

$$|{\Delta }_t|\le \frac{\rho }{1-\rho }{w}_{\ast }^{\top }(x_{\ast }-x_t).$$

$${\Delta }_t=f_0(x_t)-w_{\ast }^{\top }{x}_t.$$

$$-\rho (f^{\ast }-f_0(x_t))\le {\Delta }_t$$

$$-\rho (f^{\ast }-w_{\ast }^{\top }{x}_t-{\Delta }_t)\le {\Delta }_t$$

$$-\rho (w_{\ast }^{\top }{x}_{\ast }-w_{\ast }^{\top }{x}_t-{\Delta }_t)\le {\Delta }_t$$

$$\frac{-\rho }{1-\rho }(w_{\ast }^{\top }{x}_{\ast }-w_{\ast }^{\top }{x}_t)\le {\Delta }_t$$

$$r_t\le \frac{2\sqrt{{\beta }_t}{u}_t}{1-\rho }.$$

$$r_t$$

$$=w_{\ast }^{\top }{x}_{\ast }-(w_{\ast }^{\top }{x}_t+\Delta (x_t))$$

$$\le w_{\ast }^{\top }{x}_{\ast }-w_{\ast }^{\top }{x}_t+\rho (f^{\ast }-f_0(x_t))$$

$$=w_{\ast }^{\top }{x}_{\ast }-w_{\ast }^{\top }{x}_t+\rho r_t,$$

$$r_t$$

$$\sum _{t=1}^{T-1}{r}_t^2\le \frac{8{\beta }_{T-1}d}{(1-\rho {)}^2}\log \left(1+\frac{T{C}_b^2}{d\lambda }\right).$$

$$\sum _{t=1}^{T-1}{r}_t^2$$

$$\le \frac{4{\beta }_{T-1}}{(1-\rho {)}^2}\sum _{t=1}^{T-1}{u}_t^2$$

$$\le \frac{4{\beta }_{T-1}}{(1-\rho {)}^2}\sum _{t=0}^{T-1}{u}_t^2$$

Full text

1 Introduction

Stochastic linear bandit is a classical problem of online learning and decision-making with many influential applications, e.g., A/B testing [Claeys et al., 2021], recommendation systems [Chu et al., 2011], advertisement placements [Wang et al., 2021], clinical trials [Moradipari et al., 2020], hyperparameter tuning [Alieva et al., 2021], and new material discovery [Katz-Samuels et al., 2020].

More formally, stochastic bandit is a sequential game between an agent who chooses a sequence of actions $x_{0},...,x_{T-1}\in\mathcal{X}$ and nature who decides on a sequence of noisy observations (rewards) $y_{0},...,y_{T-1}$ according to $y_{t}=f_{0}(x_{t})+\textit{noise}$ for some underlying function $f_{0}$. The goal of the learner is to minimize the cumulative regret the agent experiences relative to an oracle who knows the best action to choose ahead of time, i.e.,

[TABLE]

where $r_{t}$ is called instantaneous regret.

Despite being highly successful in the wild, existing theory for stochastic linear bandits (or more generally learning-oracle based bandits problems [Foster et al., 2018, Foster and Rakhlin, 2020]) relies on a realizability assumption, i.e., the learner is given access to a function class $\mathcal{F}$ such that the true expected reward $f_{0}:\mathcal{X}\rightarrow\mathbb{R}$ satisfies that $f_{0}\in\mathcal{F}$. Realizability is considered one of the strongest and most restrictive assumptions in the standard statistical learning setting, but in the linear bandits, all known attempts to deviate from the realizability assumption result in a regret that grows linearly with $T$ [Ghosh et al., 2017, Lattimore et al., 2020, Zanette et al., 2020, Neu and Olkhovskaya, 2020, Bogunovic and Krause, 2021, Krishnamurthy et al., 2021].

In practical applications, it is often observed that feature-based representation of the actions with function approximations in estimating the reward can result in very strong policies even if the estimated reward functions are far from being correct [Foster et al., 2018].

So what went wrong? The critical intuition we rely on is the following:

It should be sufficient for the estimated reward function to clearly differentiate good actions from bad ones, rather than requiring it to perfectly estimate the rewards numerically.

Contributions. In this paper, we formalize this intuition by defining a new family of misspecified bandit problems based on a condition that adjusts the need for an accurate approximation pointwise at every $x\in\mathcal{X}$ according to the suboptimality gap at $x$. Unlike the existing misspecified linear bandits problems with a linear regret, our problem admits a nearly optimal $\tilde{O}(\sqrt{T})$ regret despite being heavily misspecified. Specifically:

• •

We define $\rho$-gap-adjusted misspecified ($\rho$-GAM) function approximations and characterize how they preserve important properties of the true function that are relevant for optimization.

• •

We show that the classical LinUCB algorithm [Abbasi-yadkori et al., 2011] can be used as is (up to some mild hyperparameters) to achieve an $\tilde{O}(\sqrt{T})$ regret under a moderate level of gap-adjusted misspecification ($\rho\leq O(1/\sqrt{\log T})$). In comparison, the regret bound one can obtain under the corresponding uniform misspecification setting is only $\tilde{O}(T/\sqrt{\log T})$. This represents an exponential improvement in the average regret metric $R_{T}/T$.

To the best of our knowledge, the suboptimality gap-adjusted misspecification problem was not studied before and we are the first to obtain $\sqrt{T}$-style regrets without a realizability assumption.

Technical novelty. Due to misspecification, we have technical challenges that appear in bounding the instantaneous regret and parameter uncertainty region. We tackle the challenges by a self-bounding trick, i.e., bounding the instantaneous regret by the instantaneous regret itself, which can be of independent interest in more settings, e.g., Gaussian process bandit optimization and reinforcement learning.

2 Related Work

The problem of linear bandits was first introduced in Abe and Long [1999]. Then Auer et al. [2002] proposed the upper confidence bound to study linear bandits where the number of actions is finite. Based on it, Dani et al. [2008] proposed an algorithm based on confidence ellipsoids and then Abbasi-yadkori et al. [2011] simplified the proof with a novel self-normalized martingale bound. Later Chu et al. [2011] proposed a simpler and more robust linear bandit algorithm and showed $\tilde{O}(\sqrt{dT})$ regret cannot be improved beyond a polylog factor. Li et al. [2019] further improved the regret upper and lower bound, which characterized the minimax regret up to an iterated logarithmic factor. See Lattimore and Szepesvári [2020] for a detailed survey of linear bandits.

In terms of misspecification, Ghosh et al. [2017] first studied the misspecified linear bandit with a fixed action set. They found that LinUCB [Abbasi-yadkori et al., 2011] is not robust when misspecification is large. They showed that in a favourable case when one can test the linearity of the reward function, their RLB algorithm is able to switch between the linear bandit algorithm and finite-armed bandit algorithm to address misspecification issue and achieve the $\tilde{O}(\min\{\sqrt{K},d\}\sqrt{T})$ regret where $K$ is number of arms.

The most studied setting of model misspecification is uniform misspecification where the $\ell_{\infty}$ distance between the best-in-class function and the true function is always upper bounded by some parameter $\epsilon$, i.e.,

Definition 1** ($\epsilon$-uniform misspecification).**

We say function class $\mathcal{F}$ is an $\epsilon$-uniform misspecified approximation of $f_{0}$ if there exists $f\in\mathcal{F}$ such that $\sup_{x\in\mathcal{X}}|f(x)-f_{0}(x)|\leq\epsilon$.

Under this definition, Lattimore et al. [2020] proposed the optimal design-based phased elimination algorithm for misspecified linear bandits and achieved $\tilde{O}(d\sqrt{T}+\epsilon\sqrt{d}T)$ regret when number of actions is infinite. They also found that with modified confidence band in LinUCB, LinUCB is able to achieve the same regret. With the same misspecification model, Foster and Rakhlin [2020] studied contextual bandit with regression oracle, Neu and Olkhovskaya [2020] studied multi-armed linear contextual bandit, and Zanette et al. [2020] studied misspecified contextual linear bandits after reduction of the algorithm. All of their results suffer from linear regrets. Later Bogunovic and Krause [2021] studied misspecified Gaussian process bandit optimization problem and achieved $\tilde{O}(d\sqrt{T}+\epsilon\sqrt{d}T)$ regret when linear kernel is used in Gaussian process. Moreover, their lower bound shows that $\tilde{\Omega}(\epsilon T)$ term is unavoidable in this setting.

Besides uniform misspecification, there are some work considering different definitions of misspecification. Krishnamurthy et al. [2021] defines misspecification error as an expected squared error between true function and best-in-class function where expectation is taken over distribution of context space and action space. Foster et al. [2020] considered average misspecification, which is weaker than uniform misspecification and allows tighter regret bound. However, they also have linear regrets. Our work is different from all related work mentioned above because we are working under a newly defined misspecifiation condition and show that LinUCB is a no-regret algorithm in this case.

Model misspecification is naturally addressed in the related agnostic contextual bandits setting [Agarwal et al., 2014], but these approaches typically require the action space to be finite, thus not directly applicable to our problem. In addition, empirical evidence [Foster et al., 2018] suggests that the regression oracle approach works better in practice than the agnostic approach even if realizability cannot be verified.

3 Preliminaries

3.1 Notations

Let $[n]$ denote the integer set $\{1,2,...,n\}$. The algorithm runs in $T$ rounds in total. Let $f_{0}$ denote the true function, so the maximum function value is defined as $f^{*}=\max_{x\in\mathcal{X}}f_{0}(x)$ and the maximum point is defined as $x^{*}=\mathop{\mathrm{argmax}}_{x\in\mathcal{X}}f_{0}(x)$. Let $\mathcal{X}\subset\mathbb{R}^{d}$ and $\mathcal{Y}\subset\mathbb{R}$ denote the domain and range of $f_{0}$. We use $\mathcal{W}$ to denote the parameter class of a family of linear functions $\mathcal{F}:=\{f_{w}:\mathcal{X}\rightarrow\mathcal{Y}|w\in\mathcal{W}\}$ where $f_{w}(x)=w^{\top}x$. Define $w_{*}$ as the parameter of best linear approximation function. $\|w\|_{2}\leq C_{w},\forall w\in\mathcal{W}$ and $\|x\|_{2}\leq C_{b},\forall x\in\mathcal{X}$. For a vector $x$, its $\ell_{2}$ norm is denoted by $\|x\|_{2}=\sqrt{\sum_{i=1}^{d}x^{2}_{i}}$ and for a matrix $A$ its operator norm is denoted by $\|A\|_{\mathrm{op}}$. For a vector $x$ and a square matrix $A$, define $\|x\|^{2}_{A}=x^{\top}Ax$.

3.2 Problem Setup

We consider the following optimization problem:

[TABLE]

where $f_{0}$ is the true function which might not be linear in $\mathcal{X}$. We want to use a linear function $f_{w}=w^{\top}x\in\mathcal{F}$ to approximate $f_{0}$ and maximize $f_{0}$. At time $0\leq t\leq T-1$, after querying a data point $x_{t}$, we will receive a noisy feedback:

[TABLE]

where $\eta_{t}$ is independent, zero-mean, and $\sigma$-sub-Gaussian noise.

The major highlight of our study is that we do not rely on the popular realizability assumption (i.e. $f_{0}\in\mathcal{F}$) that is frequently assumed in the existing function approximation literature. Alternatively, we propose the following gap-adjusted misspecification condition.

Definition 2** ($\rho$-gap-adjusted misspecification).**

We say a function $f$ is a $\rho$-gap-adjusted misspecified (or $\rho$-GAM in short) approximation of $f_{0}$ if for parameter $0\leq\rho<1$,

[TABLE]

We say function class $\mathcal{F}=\{f_{w}|w\in\mathcal{W}\}$ satisfies $\rho$-GAM for $f_{0}$, if there exists $w^{*}\in\mathcal{W}$ such that $f_{w_{*}}$ is a $\rho$-GAM approximation of $f_{0}$.

Observe that when $\rho=0$, this recovers the standard realizability assumption, but when $\rho>0$ it could cover many misspecified function classes.

Figure 1(a) shows a 1-dimensional example with $f_{w}(x)=0.75x+0.5$ and piece-wise linear function $f_{0}(x)$ that satisfies local misspecification. With Definition 2, we have the following proposition.

Proposition 3**.**

Let $f$ be a $\rho$-GAM approximation of $f_{0}$ (Definition 2). Then it holds:

• •

(Preservation of maximizers)

[TABLE]

• •

(Preservation of max value)

[TABLE]

• •

(Self-bounding property)

[TABLE]

This tells $f$ and $f_{0}$ coincide on the same global maximum points and the same global maxima if Definition 2 is satisfied, while allowing $f$ and $f_{0}$ to be different (potentially large) at other locations. Therefore, Definition 2 is a “local” assumption that does not require $f$ to be uniformly close to $f_{0}$ (e.g. the “uniform” misspecification assumes $\sup_{x\in\mathcal{X}}|f(x)-f_{0}(x)|\leq\rho$). Proof of Proposition 3 is shown in Appendix A.

In addition, we can modify Definition 2 with a slightly weaker condition that only requires $\mathop{\mathrm{argmax}}_{x}f(x)=\mathop{\mathrm{argmax}}_{x}f_{0}(x)$ but not necessarily $\max_{x\in\mathcal{X}}f(x)=f^{*}$.

Definition 4** (Weak $\rho$-gap-adjusted misspecification).**

Denote $f_{w}^{*}=\max_{x\in\mathcal{X}}f(x)$. Then we say $f$ is (weak) $\rho$-gap-adjusted misspecification approximation of $f_{0}$ for a parameter $0\leq\rho<1$ if:

[TABLE]

See Figure 1(b) for an example satisfying Definition 4, in which there is a constant gap between $f^{*}_{w}$ and $f^{*}$. The idea of this weaker assumption is that we can always extend the function class by adding a single offset parameter $c$ w.l.o.g. to learn the constant gap $f^{*}-f_{w}^{*}$. In the linear case, this amounts to homogenizing the feature vector by appending $1$. For this reason, we stick to Definition 2 and linear function approximation for conciseness and clarity in the main paper. See Appendix B for formal statements and Appendix C for proofs of regret bound of linear bandits under Definition 4.

Note that both Definition 2 and Definition 4 are defined generically which do not require any assumptions on the parametric form of $f$. While we focus on the linear bandit setting in this paper, this notion can be considered for arbitrary function approximation learning problems.

3.3 Assumptions

Assumption 5** (Boundedness).**

For any $x\in\mathcal{X}$, $\|x\|_{2}\leq C_{b}$. For any $w\in\mathcal{W}$, $\|w\|_{2}\leq C_{w}$. Moreover, for any $x,\tilde{x}\in\mathcal{X}$, the true expected reward function $|f_{0}(x)-f_{0}(\tilde{x})|\leq F$.

These are mild assumptions that we assume for convenience. Relaxations of these are possible but not the focus of this paper. Note that the additional assumption is not required when $f_{0}$ is realizable.

Assumption 6**.**

Suppose $\mathcal{X}\in\mathbb{R}^{d}$ is a compact set, and all the global maximizers of $f_{0}$ live on the $d-1$ dimensional hyperplane. i.e., $\exists a\in\mathbb{R}^{d},b\in\mathbb{R}^{1}$, s.t.

[TABLE]

For instance, when $d=1$, the above reduces to that $f_{0}$ has a unique maximizer. This is a compatibility assumption for Definition 2, since any linear function that violates Assumption 6 will not satisfy Definition 2.

In addition, to obtain an $\tilde{O}(\sqrt{T})$ regret, for any finite sample $T$, we require the following condition.

Assumption 7** (Low misspecification).**

The linear function class is a $\rho$-GAM approximation of $f_{0}$ with

[TABLE]

The condition is required for technical reasons. Relaxing this condition for LinUCB may require fundamental breakthroughs that knock out logarithmic factors from its regret analysis. This will be further clarified in the proof. In general, however, we conjecture that this condition is not needed and there are algorithms that can achieve $\tilde{O}(\sqrt{T}/(1-\rho))$ regret for any $\rho<1$, but a new algorithm needs to be designed.

While this assumption may suggest that we still require realizability in a truly asymptotic world, handling a $O(1/\sqrt{\log T})$ level of misspecification is highly non-trivial in finite sample setting. For instance, if $T$ is a trillion, $1/\sqrt{\log(1e12)}\approx 0.19$. This means that for most practical cases, LinUCB is able to tolerate a constant level of misspecification under the GAM model.

3.4 LinUCB Algorithm

We will focus on analyzing the classical Linear Upper Confidence Bound (LinUCB) algorithm due to [Dani et al., 2008, Abbasi-yadkori et al., 2011], shown below.

4 Main Results

In this section, we show that the classical LinUCB algorithm [Abbasi-yadkori et al., 2011] works in $\rho$-gap-adjusted misspecified linear bandits and achieves cumulative regret at the order of $\tilde{O}(\sqrt{T}/(1-\rho))$. The following theorem shows the cumulative regret bound.

Theorem 8**.**

Suppose Assumptions 5, 6, and 7 hold. Set

[TABLE]

Then Algorithm 1 guarantees w.p. $>1-\delta$ simultaneously for all $T=1,2,...$

[TABLE]

Remark 9**.**

The result shows that LinUCB achieves $\tilde{O}(\sqrt{T})$ cumulative regret bound and thus it is a no-regret algorithm in $\rho$-gap-adjusted misspecified linear bandits. In contrast, LinUCB can only achieve $\tilde{O}(\sqrt{T}+\epsilon T)$ regret in uniformly misspecified linear bandits. Even if $\epsilon=\tilde{O}(1/\sqrt{\log T})$, the resulting regret $\tilde{O}(T/\sqrt{\log T})$ is still exponentially worse than ours.

Proof.

By definition of cumulative regret, function range absolute bound $F$, and Cauchy-Schwarz inequality,

[TABLE]

Observe that the choice of $\beta_{t}$ is monotonically increasing in $t$. Also by Lemma 14, we get that with probability $1-\delta$, $w_{*}\in\text{Ball}_{t},\forall t=1,2,3,...$, which verifies the condition to apply Lemma 12 simultaneously for all $T=1,2,3,...$, thereby completing the proof. ∎

4.1 Regret Analysis

The proof follows the LinUCB analysis closely. The main innovation is a self-bounding argument that controls the regret due to misspecification by the regret itself. This appears in Lemma 11 and then again in the proof of Lemma 14.

Before we proceed, let $\Delta_{t}$ denote the deviation term of our linear function from the true function at $x_{t}$, formally,

[TABLE]

And our observation model (eq. (1)) becomes

[TABLE]

Moreover, we have the following lemma showing the property of deviation term $\Delta_{t}$.

Lemma 10** (Bound of deviation term).**

$\forall t\in\{0,1,\ldots,T-1\}$,

[TABLE]

Proof.

Recall the definition of deviation term in eq. (6):

[TABLE]

By Definition 2, $\forall t\in\{0,1,\ldots,T-1\}$,

[TABLE]

where the third line is by Proposition 3 and the proof completes by taking the absolute value of the lower and upper bounds. ∎

Next, we prove instantaneous regret bound and its sum of squared regret version in the following two lemmas:

Lemma 11** (Instantaneous regret bound).**

Define $u_{t}:=\|x_{t}\|_{\Sigma_{t}^{-1}}$, assume $w_{*}\in\mathrm{Ball}_{t}$ then for each $t\geq 1$

[TABLE]

Proof.

By definition of instantaneous regret,

[TABLE]

where the inequality is by Definition 2. Therefore, by rearranging the inequality we have

[TABLE]

where the last inequality is by Lemma 13. ∎

Lemma 12**.**

Assume $\beta_{t}$ is monotonically nondecreasing and $w_{*}\in\mathrm{Ball}_{t}$ for all $t=1,...,T-1$, then

[TABLE]

Proof.

By definition $u_{t}=\sqrt{x^{\top}_{t}\Sigma^{-1}_{t}x_{t}}$ and Lemma 11,

[TABLE]

where the second inequality is by the monotonic increasing property of $\beta_{t}$ and the last inequality uses the elliptical potential lemma (Lemma 16). ∎

Previous two lemmas hold on the following lemma, bounding the gap between $f^{*}$ and the linear function value at $x_{t}$, shown below.

Lemma 13**.**

Define $u_{t}=\|x_{t}\|_{\Sigma_{t}^{-1}}$ and assume $\beta_{t}$ is chosen such that $w_{*}\in\mathrm{Ball}_{t}$. Then

[TABLE]

Proof.

Let $\tilde{w}$ denote the parameter that achieves $\mathop{\mathrm{argmax}}_{w\in\mathrm{Ball}_{t}}w^{\top}x_{t}$, by the optimality of $x_{t}$,

[TABLE]

where the second inequality applies Holder’s inequality; the last line uses the definition of $\mathrm{Ball}_{t}$ (note that both $w_{*},\tilde{w}\in\mathrm{Ball}_{t}).$ ∎

4.2 Confidence Analysis

All analysis in the previous section requires $w_{*}\in\mathrm{Ball}_{t},\forall t\in[T]$. In this section, we show that our choice of $\beta_{t}$ in (5) is valid and $w_{*}$ is trapped in the uncertainty set $\mathrm{Ball}_{t}$ with high probability.

Lemma 14** (Feasibility of $\mathrm{Ball}_{t}$).**

Suppose Assumptions 5, 6, and 7 hold. Set $\beta_{t}$ as in eq. (5). Then, w.p. $>1-\delta$,

[TABLE]

Proof.

By setting the gradient of objective function in eq. (4) to be [math], we obtain the closed form solution of eq. (4):

[TABLE]

Therefore,

[TABLE]

where the second equation is by eq. (7) and the first two terms of eq. (8) can be further simplified as

[TABLE]

where the second equation is by definition of $\Sigma_{t}$ (eq. (3)). Therefore, eq. (8) can be rewritten as

[TABLE]

Multiply both sides by $\Sigma_{t}^{\frac{1}{2}}$ and we have

[TABLE]

Take a square of both sides and apply generalized triangle inequality, we have

[TABLE]

The remaining task is to bound these three terms separately. The first term of eq. (9) is bounded as

[TABLE]

where the first inequality is by definition of $\Sigma_{t}$ and $\|\Sigma^{-1}_{t}\|_{\mathrm{op}}\leq 1/\lambda$ and the second inequality is by choice of $\lambda=\sigma^{2}/C^{2}_{w}$.

The second term of eq. (9) can be bounded by Lemma 17 and Lemma 20:

[TABLE]

where $\delta_{t}$ is chosen as $3\delta/(\pi^{2}t^{2})$ so that the total failure probabilities over $T$ rounds can always be bounded by $\delta/2$:

[TABLE]

And the third term of eq. (9) can be bounded as

[TABLE]

where the last line is by taking the absolute value and Cauchy-Schwarz inequality. Continue the proof and we have

[TABLE]

where the first inequality is due to Cauchy-Schwarz inequality and the second uses the self-bounding properties $|\Delta_{i}|\leq\rho r_{i}$ from Proposition 3 and Lemma 15.

To put things together, we have shown that w.p. $>1-\delta$, for any $t\geq 1$,

[TABLE]

where we condition on (10) for the rest of the proof.

Observe that this implies that the feasibility of $w_{*}$ in $\mathrm{Ball}_{t}$ can be enforced if we choose $\beta_{t}$ to be larger than (10). The feasiblity of $w_{*}$ in turn allows us to apply Lemma 11 to bound the RHS with $\beta_{0},...,\beta_{t-1}$. We will use induction to prove that our choice

[TABLE]

is valid, where short hand

[TABLE]

For the base case $t=1$, by eq. (10) and the definition of $\beta_{1}$ we directly have $\|\hat{w}_{1}-w_{*}\|^{2}_{\Sigma_{1}^{-1}}\leq\beta_{1}$. Assume our choice of $\beta_{i}$ is feasible for $i=1,...,t-1$, then we can write

[TABLE]

where the second line is due to non-decreasing property of $\beta_{t}$. Then by Lemma 16 and Assumption 7, we have

[TABLE]

The critical difference from the standard LinUCB analysis here is that if $\beta_{t-1}$ appears on the LHS of the bound and if its coefficient is larger, any valid bound for $\beta_{t}$ will have to grow exponentially in $t$. This is where Assumption 7 helps us. Assumption 7 ensures that the coefficient of $\beta_{t-1}$ is smaller than $1/2$, so we can take $\beta_{t-1}\leq\beta_{t}$ and move $\beta_{t}/2$ to the right-hand side. ∎

Proof of previous lemma needs the following two lemmas.

Lemma 15** (Upper bound of $\sum_{i=0}^{t-1}x^{\top}_{i}\Sigma_{t}^{-1}x_{i}$).**

[TABLE]

Proof.

Recall that $\Sigma_{t}=\sum_{i=0}^{t-1}x_{i}x_{i}^{T}+\lambda I_{d}$.

[TABLE]

The last line follows from the fact that $\Sigma^{-1}_{t}$ is positive semidefinite. ∎

Lemma 16** (Upper bound of $\sum_{i=0}^{t-1}x^{\top}_{i}\Sigma_{i}^{-1}x_{i}$ (adapted from Abbasi-yadkori et al. [2011])).**

[TABLE]

Proof.

First we prove that $\forall i\in\{0,1,...,t-1\},0\leq x_{i}^{\top}\Sigma^{-1}_{i}x_{i}<1$. Recall the definition of $\Sigma_{i}$ and we know $\Sigma^{-1}_{i}$ is a positive semidefinite matrix and thus $0\leq x_{i}^{\top}\Sigma^{-1}_{i}x_{i}$. To prove $x_{i}^{\top}\Sigma^{-1}_{i}x_{i}<1$, we need to decompose $\Sigma_{i}$ and write

[TABLE]

Let $A=-x_{i}x_{i}^{\top}+\lambda I+\sum_{j=0}^{i-1}x_{j}x^{\top}_{j}$ and it becomes

[TABLE]

By Sherman-Morrison lemma (Lemma 18), we have

[TABLE]

Next we use the fact that $\forall x\in[0,1),x\leq 2\log(x+1)$ and we have

[TABLE]

where the last two lines are by Lemma 19 and Lemma 20. ∎

5 Technical Lemmas

Lemma 17** (Self-normalized bound for vector-valued martingales (Lemma A.9 of Agarwal et al. [2021])).**

Let $\{\eta_{i}\}_{i=1}^{\infty}$ be a real-valued stochastic process with corresponding filtration $\{\mathcal{F}_{i}\}_{i=1}^{\infty}$ such that $\eta_{i}$ is $\mathcal{F}_{i}$ measurable, $\mathbb{E}[\eta_{i}|\mathcal{F}_{i-1}]=0$, and $\eta_{i}$ is conditionally $\sigma$-sub-Gaussian with $\sigma\in\mathbb{R}^{+}$. Let $\{X_{i}\}_{i=1}^{\infty}$ be a stochastic process with $X_{i}\in\mathcal{H}$ (some Hilbert space) and $X_{i}$ being $\mathcal{F}_{t}$ measurable. Assume that a linear operator $\Sigma:\mathcal{H}\rightarrow\mathcal{H}$ is positive definite, i.e., $x^{\top}\Sigma x>0$ for any $x\in\mathcal{H}$. For any $t$, define the linear operator $\Sigma_{t}=\Sigma_{0}+\sum_{i=1}^{t}X_{i}X^{\top}_{i}$ (here $xx^{\top}$ denotes outer-product in $\mathcal{H}$). With probability at least $1-\delta$, we have for all $t\geq 1$:

[TABLE]

Lemma 18** (Sherman-Morrison lemma [Sherman and Morrison, 1950]).**

Let $A$ denote a matrix and $b,c$ denote two vectors. Then

[TABLE]

Lemma 19** (Lemma 6.10 of Agarwal et al. [2021]).**

Define $u_{t}=\sqrt{x^{\top}_{t}\Sigma^{-1}_{t}x_{t}}$ and we have

[TABLE]

Lemma 20** (Potential function bound (Lemma 6.11 of Agarwal et al. [2021])).**

For any sequence $x_{0},...,x_{T-1}$ such that for $t<T,\|x_{t}\|_{2}\leq C_{b}$, we have

[TABLE]

6 Conclusion

We study linear bandits with the underlying reward function being non-linear, which falls into the misspecified bandit framework. Existing work on misspecified bandit usually assumes uniform misspecification where the $\ell_{\infty}$ distance between the best-in-class function and the true function is upper bounded by the misspecification parameter $\epsilon$. Existing lower bound shows that the $\tilde{\Omega}(\epsilon T)$ term is unavoidable where $T$ is the time horizon, thus the regret bound is always linear. However, in solving optimization problems, one only cares about the approximation error near the global optimal point and approximation error is allowed to be large in highly suboptimal regions. In this paper, we capture this intuition and define a natural model of misspecification, called $\rho$-gap-adjusted misspecificaiton, which only requires the approximation error at each input $x$ to be proportional to the suboptimality gap at $x$ with $\rho$ being the proportion parameter.

Previous work found that classical LinUCB algorithm is not robust in $\epsilon$-uniform misspecified linear bandit when $\epsilon$ is large. However, we show that LinUCB is automatically robust against such gap-adjusted misspecification. Under mild conditions, e.g., $\rho\leq O(1/\sqrt{\log T})$, we prove that it achieves the near-optimal $\tilde{O}(\sqrt{T})$ regret for problems that the best-known regret is almost linear. Also, LinUCB doesn’t need the knowledge of $\rho$ to run. However, if the upper bound of $\rho$ is revealed to LinUCB, the $\beta_{t}$ term can be carefully chosen according to eq. (11). Our technical novelty lies in a new self-bounding argument that bounds part of the regret due to misspecification by the regret itself, which can be of independent interest in more settings.

We believe our analysis for LinUCB is tight and the requirement that $\rho=O(1/\sqrt{\log T})$ is essential, but we conjecture that there is a different algorithm that could handle constant $\rho$ or even when $\rho$ approaches $1$ at a rate of $O(1/\sqrt{T})$. We leave the resolution to this conjecture as future work. For completeness, we include a simulation section in Appendix D.

More broadly, our paper opens a brand new door for research in model misspecification, including misspecified linear bandits, misspecified kernelized bandits, and even reinforcement learning with misspecified function approximation. Moreover, we hope our paper make people rethink about the relationship between function optimization and function approximation. In the future, much more can be done. For example, we can design a new no-regret algorithm that works under gap-adjusted misspecification framework where $\rho$ is a constant, and study $\rho$-gap-adjusted misspecified Gaussian process bandit optimization.

Acknowledgments

The work was partially supported by NSF Awards #2007117 and #2003257. We thank Ilija Bogunovic for the discussion at the early stage of this paper. Finally, we thank UAI reviewers and the area chair for their valuable input that led to improvements to the paper.

Appendix A Proof of Proposition 3

Equivalently, $\rho$-gap-adjusted misspecification (Definition 2) satisfies

[TABLE]

Proof of preservation of max value: $\max_{x\in\mathcal{X}}f(x)=f^{*}$.

Let $f^{*}_{w}:=\max_{x\in\mathcal{X}}f(x)$. We first prove $f^{*}_{w}\leq f^{*}$ by contradiction. Suppose $f^{*}_{w}>f^{*}$, since $\mathcal{X}$ is compact, there exists $x_{w}\in\mathcal{X}$ such that $f(x_{w})=f^{*}_{w}>f^{*}$. Then by eq. (12) this implies

[TABLE]

Contraction! Therefore, $f_{w}^{*}\leq f^{*}$. On the other hand, choose $x_{0}\in\mathop{\mathrm{argmax}}_{x\in\mathcal{X}}f_{0}(x)$, then by (12) $f(x_{0})=f_{0}(x_{0})=f^{*}$. This implies $f_{w}^{*}\geq f^{*}$. Combing both results to obtain $f_{w}^{*}=f^{*}$. ∎

Proof of preservation of maximizers: $\mathop{\mathrm{argmax}}_{x}f(x)=\mathop{\mathrm{argmax}}_{x}f_{0}(x)$.

Using that $f(x)\leq\rho f^{*}+(1-\rho)f_{0}(x)$ and $\max_{x\in\mathcal{X}}f(x)=f^{*}$, it is easy to verify $\mathop{\mathrm{argmax}}_{x}f(x)\subset\mathop{\mathrm{argmax}}_{x}f_{0}(x)$. On the other hand, if $x^{\prime}\in\mathop{\mathrm{argmax}}_{x}f_{0}(x)$, then by eq. (12) $f(x^{\prime})=f_{0}(x^{\prime})=f^{*}$ and this means $\mathop{\mathrm{argmax}}_{x}f_{0}(x)\subset\mathop{\mathrm{argmax}}_{x}f(x)$. ∎

Proof of self-bounding property.

This directly comes from the definition. ∎

Appendix B Property of Weak $\rho$-Gap-Adjusted Misspecification

First we recall Definition 4.

Definition 21** (Restatement of Weak $\rho$-gap-adjusted misspecification).**

Denote $f_{w}^{*}=\max_{x\in\mathcal{X}}f(x)$. Then we say $f$ is (weak) $\rho$-gap-adjusted misspecification approximation of $f_{0}$ for a parameter $0\leq\rho<1$ if:

[TABLE]

Under the weak $\rho$-gap-adjusted misspecification condition, it no longer holds $f_{w}^{*}=f^{*}$. However, it still preserves the maximizers.

Proposition 22**.**

Under the weak $\rho$-gap-adjusted misspecification condition, it holds

[TABLE]

Proof.

Suppose $x^{\prime}\in\mathop{\mathrm{argmax}}_{x}f(x)$, then by definition

[TABLE]

On the other hand, if $x^{\prime}\in\mathop{\mathrm{argmax}}_{x}f_{0}(x)$, then

[TABLE]

∎

The next proposition shows the weak $\rho$-adjusted misspecification condition characterizes the suboptimality gap between $f$ and $f_{0}$.

Proposition 23**.**

Denote $g(x):=f^{*}_{w}-f(x)\geq 0$, $g_{0}(x):=f^{*}-f_{0}(x)\geq 0$, then the weak $\rho$-gap-adjusted misspecification condition implies:

[TABLE]

This can be proved directly by the triangular inequality. This reveals the weak $\rho$-gap-adjusted misspecification condition requires $g(x)$ to live in the band $[(1-\rho)g_{0}(x),(1+\rho)g_{0}(x)]$, and the concrete maximum values $f_{w}^{*}$ and $f^{*}$ can be arbitrarily different.

Appendix C Linear Bandits under the Weak $\rho$-Gap-Adjusted Misspecification

We need to slightly modify LinUCB [Abbasi-yadkori et al., 2011] and work with the following LinUCBw algorithm.

Theorem 24**.**

Suppose Assumptions 5, 6, and 7 hold. W.l.o.g., assuming $c^{*}=f^{*}-f_{w}^{*}\leq F$. Set

[TABLE]

Then Algorithm 2 guarantees w.p. $>1-\delta$ simultaneously for all $T=1,2,...$

[TABLE]

Remark 25**.**

The result again shows that LinUCBw algorithm achieves $\tilde{O}(\sqrt{T})$ cumulative regret and thus it is also a no-regret algorithm under the weaker condition (Definition 4). Note Definition 4 is quite weak which even doesn’t require the true function sits within the approximation function class.

Proof.

The analysis is similar to the $\rho$-gap-adjusted case but includes $c^{*}=f^{*}-f^{*}_{w}$. For instance, let $\Delta^{w}_{t}$ denote the deviation term of our linear function from the true function at $x_{t}$, then

[TABLE]

And our observation model (eq. (1)) becomes

[TABLE]

Then similar to Lemma 10, we have the following lemma, whose proof is nearly identical to Lemma 10.

Lemma 26** (Bound of deviation term).**

$\forall t\in\{0,1,\ldots,T-1\}$,

[TABLE]

We also provide the following lemma, which is the counterpart of Lemma 13.

Lemma 27**.**

Define $u_{t}=\left\|\begin{bmatrix}x_{t}\\ 1\end{bmatrix}\right\|_{\Sigma_{t}^{-1}}$ and assume $\beta_{t}$ is chosen such that $w_{*}\in\mathrm{Ball}_{t}$. Then

[TABLE]

Proof.

Let $\tilde{w},\tilde{c}$ denote the parameter that achieves $\mathop{\mathrm{argmax}}_{w,c\in\mathrm{Ball}_{t}}w^{\top}x_{t}+c$, by the optimality of $x_{t}$,

[TABLE]

where the second inequality applies Holder’s inequality; the last line uses the definition of $\mathrm{Ball}_{t}$ (note that both $\begin{bmatrix}\tilde{w}^{\top},\tilde{c}\end{bmatrix},\begin{bmatrix}w_{*}^{\top},c^{*}\end{bmatrix}\in\mathrm{Ball}_{t}).$ ∎

The rest of the analysis follows the analysis of Theorem 8. ∎

Appendix D Simulation

In this section, we run a simulation on a $1$-dimensional test function shown in Figure 2(a). Here we run the first $10$ iterations with uniform sampling and the remaining $100$ iterations are using LinUCB algorithm. In Figure 2(b) we can see that cumulative regret is increasing with uniform sampling but it doesn’t increase when running LinUCB. The reason behind it is that under the gap-adjusted misspecification, LinUCB is able to quickly find the optimal point $x_{*}=0$.