Contextual Linear Bandits under Noisy Features: Towards Bayesian Oracles

TL;DR

This paper investigates contextual linear bandit problems with noisy and incomplete features, proposing a Bayesian-based algorithm that achieves near-optimal regret bounds despite feature uncertainty.

Contribution

It introduces a Bayesian analysis for noisy feature bandits and develops an algorithm that approximates the Bayesian oracle, achieving $ ilde{O}(d ext{ } ext{sqrt}(T))$ regret.

Findings

Bayesian analysis reveals significant deviations from classical assumptions due to noise.

The proposed algorithm attains near-optimal regret bounds in noisy feature settings.

Experimental results on synthetic and real datasets validate the approach.

Abstract

We study contextual linear bandit problems under feature uncertainty, where the features are noisy and have missing entries. To address the challenges posed by this noise, we analyze Bayesian oracles given the observed noisy features. Our Bayesian analysis reveals that the optimal hypothesis can significantly deviate from the underlying realizability function, depending on the noise characteristics. These deviations are highly non-intuitive and do not occur in classical noiseless setups. This implies that classical approaches cannot guarantee a non-trivial regret bound. Therefore, we propose an algorithm that aims to approximate the Bayesian oracle based on the observed information under this model, achieving $\tilde{O}(d\sqrt{T})$ regret bound when there is a large number of arms. We demonstrate the proposed algorithm using synthetic and real-world datasets.

Figures

Equations

$${\mathcal{A}}_t=\left\{a\in \mathcal{A};\Vert x_{a,t}{\Vert }_2=O\left(\sqrt{\Vert m_{a,t}{\Vert }_2^2\log (KT)}\right)\right\}.$$

$$\Vert x_{a,t}{\Vert }_2=O\left(\sqrt{\Vert m_{a,t}{\Vert }_2^2\log (KT)}\right),$$

$$a_t^{\star }$$

$$=\arg \underset{a\in {\mathcal{A}}_t}{\max }\mathbb{E}\left[z_{a,t}^{\top }{\theta }^{\star }|x_{a,t}\right],$$

$$R(T)=\sum _{t=1}^T\mathbb{E}\left[z_{a_t^{\star },t}^{\top }{\theta }^{\star }-z_{a_t,t}^{\top }{\theta }^{\star }\right].$$

$${\theta }^{\prime }:=({\Sigma }_f+{\Sigma }_n{)}^{-1}{\Sigma }_f{\theta }^{\star }\text{ and }\overline{\theta }:=[{\nu }_f^{\top }{\theta }^{\star }-{\nu }_f^{\top }{\theta }^{\prime };{\theta }^{\prime }].$$

$$a_t^{\star }\in \arg \underset{a\in {\mathcal{A}}_t}{\max }\mathbb{E}[z_{a,t}^{\top }{\theta }^{\star }|x_{a,t}]$$

$$=\arg \underset{a\in {\mathcal{A}}_t}{\max }{x}_{a,t}^{\top }{\theta }^{\prime }.$$

$$a_t^{\star }$$

$$=\arg \underset{a\in {\mathcal{A}}_t}{\max }\mathbb{E}[z_{a,t}|x_{a,t}{]}^{\top }{\theta }^{\star }.$$

$$\mathbb{E}[z_{a,t}|x_{a,t}{]}^{\top }{\theta }^{\star }$$

$$=x_{a,t}^{\top }{\theta }^{\prime }-{\nu }_f^{\top }{\theta }^{\prime }+{\nu }_f^{\top }{\theta }^{\star }=[1;x_{a,t}{]}^{\top }\overline{\theta },$$

$$R(T)$$

$$=\sum _{t=1}^T\mathbb{E}\left[[1;x_{a_t^{\star },t}{]}^{\top }\overline{\theta }-[1;x_{a_t,t}{]}^{\top }\overline{\theta }\right].$$

$${\mathcal{C}}_t=$$

$$\Vert {\hat{\theta }}_t-\theta {\Vert }_{V_t}\le C_1\sqrt{(d+1)\log ((1+t)T)}+C_2{\lambda }^{1/2}\},$$

$$\overline{x}(\nu ,\Sigma ,x{)}_{\mathcal{S}}$$

$$\overline{x}(\nu ,\Sigma ,x{)}_{\mathcal{U}}$$

$$a_t^{\star }$$

$$=\underset{a\in {\mathcal{A}}_t}{\mathrm{argmax}}[1;\overline{x}({\nu }_f,{\Sigma }_f+{\Sigma }_n,x_{a,t}){]}^{\top }\overline{\theta }$$

$$=\underset{a\in {\mathcal{A}}_t}{\mathrm{argmax}}\overline{x}({\nu }_f,{\Sigma }_f+{\Sigma }_n,x_{a,t}{)}^{\top }{\theta }^{\prime }.$$

$$\mathbb{E}[x_{a,t}^{\prime }|x_{a,t}]=\overline{x}({\nu }_f,{\Sigma }_f+{\Sigma }_n,x_{a,t}),$$

$$\mathbb{E}[z_{a,t}^{\top }{\theta }^{\star }|x_{a,t}]$$

$$=\mathbb{E}\left[(x_{a,t}^{\prime }-{\nu }_f{)}^{\top }{\theta }^{\prime }+{\nu }_f^{\top }{\theta }^{\star }|x_{a,t}\right]$$

$$=\mathbb{E}[x_{a,t}^{\prime }|x_{a,t}{]}^{\top }{\theta }^{\prime }+{\nu }_f^{\top }({\theta }^{\star }-{\theta }^{\prime })$$

$$=\overline{x}({\nu }_f,{\Sigma }_f+{\Sigma }_n,x_{a,t}{)}^{\top }{\theta }^{\prime }+{\nu }_f^{\top }({\theta }^{\star }-{\theta }^{\prime })$$

$$=[1;\overline{x}({\nu }_f,{\Sigma }_f+{\Sigma }_n,x_{a,t}){]}^{\top }\overline{\theta },$$

$$R(T)$$

$$=\sum _{t=1}^T\mathbb{E}[[1;\overline{x}({\nu }_f,{\Sigma }_f+{\Sigma }_n,x_{a_t^{\star },t}){]}^{\top }\overline{\theta }$$

$$-[1;\overline{x}({\nu }_f,{\Sigma }_f+{\Sigma }_n,x_{a_t,t}){]}^{\top }\overline{\theta }].$$

$${\mathcal{C}}_t$$

$$+C_2{\lambda }^{1/2}+C_6(d/\hat{p}{)}^{3/2}\sqrt{\frac{\log (KT)}{K}}{X}_t\},$$

$$a_t$$

$$+C_6(d/\hat{p}{)}^{3/2}\sqrt{\frac{\log (KT)}{K}}{X}_{t-1})\sqrt{{\hat{z}}_{a,t}^{\top }{V}_{t-1}^{-1}{\hat{z}}_{a,t}}\}.$$

$$R(T)=\tilde{O}\left(d\sqrt{T}+\frac{d^2}{p^{3/2}}\sqrt{\frac{T}{K}}+\frac{d}{p^{4}K}\right).$$

$$R(T)=\tilde{O}(d\sqrt{T}).$$

$$\Vert {\hat{z}}_{a_s,s}(t){\Vert }_2=O\left(\sqrt{d\log (KT)}\right).$$

$$\mathbb{E}\left[{\overline{z}}_{a_t^{\star },t}^{\top }\overline{\theta }-{\overline{z}}_{a_t,t}^{\top }\overline{\theta }\right]=O\left(\sqrt{\log (K)}\right).$$

$$\sum _{t=1}^{\tau }\mathbb{E}\left[r_t\right]=O\left(\tau \sqrt{\log (K)}\right)=\tilde{O}\left(d/(K{p}^4)\right).$$

Full text

1 INTRODUCTION

The bandit problem (Lai and Robbins, 1985) is a fundamental sequential decision-making problem when dealing with the exploration-exploitation trade-off. It has received considerable attention due to its applicability to a wide range of real-world problems, such as clinical trials (Thompson, 1933), economics (Schlag, 1998), routing (Awerbuch and Kleinberg, 2004), and ranking (Radlinski et al., 2008). In a basic multi-armed-bandit (MAB) problem, there is a finite number of actions or “arms” and in each round, an agent selects an arm and observes a random reward. The goal is to minimize regret, which is the difference in expected cumulative reward between the agent and an oracle policy that knows latent parameters.

A natural extension for the basic MAB is to provide the agent with contextual information (Langford and Zhang, 2008) that is present in many real-life problems such as personalized recommendations (Bouneffouf et al., 2012), web server defense (Jung et al., 2012), and information retrieval (Hofmann et al., 2011). In each round, the environment draws a context, and the agent observes it. Then the agent chooses an arm based on the contextual information and receives a random reward. contextual linear bandit problems include feature maps between context and arms, so each arm has a feature vector in $\mathbb{R}^{d}$. There is also a latent parameter in $\mathbb{R}^{d}$, and the mean reward for each arm follows a linear model between the latent parameter and feature vector. For contextual linear bandit problems, Auer (2002); Chu et al. (2011), and Abbasi-Yadkori et al. (2011) proposed algorithms based on the principle of optimism in the face of uncertainty.

The uncertainty of features is an important issue for many domains, including computational biology, clinical studies, and economics (Sterne et al., 2009; Troyanskaya et al., 2001; Wooldridge, 2007). Therefore, the estimation of latent parameters or learning models under noisy observations has been widely studied by Loh and Wainwright (2011); Lounici (2014); Pavez and Ortega (2020), and You et al. (2020). Recommendation systems (Li et al., 2010; Balakrishnan et al., 2018) can construct feature vectors of items by pre-processing item information such as text descriptions, categories, or figures; it is natural to have some feature noise from the pre-processing. Noisy features can even have missing entries for several reasons, including communication failure and human error. Noise can also be intentionally added to features to preserve privacy. For example, features often represent user profile information, and a recent trend in providing services that respect privacy (called differential privacy (Dwork, 2008)) is to add noise to user profile information. Therefore, noisy feature information seems natural and essential for real-world applications.

In this paper, we consider a variant of the contextual linear bandit problem in which random noise exists in the feature vectors. Here, we briefly describe these noisy features. At each time $t$, the true feature $z_{a,t}\in\mathbb{R}^{d}$ for an arm $a$ in the set of arms is generated randomly encoding context, and the mean reward is $z_{a,t}^{\top}\theta^{\star}$ where $\theta^{\star}\in\mathbb{R}^{d}$ is a latent parameter. An agent can only observe a noise feature vector $x_{a,t}\in\mathbb{R}^{d}$, rather than $z_{a,t}$, which is defined as $x_{a,t}=(z_{a,t}+\varepsilon_{a,t})\circ m_{a,t}$ where $\circ$ is the element-wise product, $\varepsilon_{a,t}\in\mathbb{R}^{d}$ is randomly generated from a Gaussian noise vector, and $m_{a,t}\in\{0,1\}^{d}$ is randomly generated from a Bernoulli distribution for missing entries, which follows the same framework as in Loh and Wainwright (2011). The missing data can be said to be missing completely at random (MCAR). To handle the noisy features, we first define an oracle policy from a Bayesian perspective given the observed noisy features. Taking insights from this oracle policy, we propose an algorithm that can achieve regret bound $\tilde{O}(d\sqrt{T})$ with respect to the feature dimension $d$ and horizon time $T$.

Related work. Auer (2002) first analyzed the linear payoff model in the bandit problem. Algorithms LinRel (Auer, 2002) and LinUCB (Chu et al., 2011) compute the expected rewards and the corresponding confidence intervals to control the exploration-exploitation trade-off and achieve an $\tilde{O}(\sqrt{dT})$ regret bound. Abbasi-Yadkori et al. (2011) considered the linear bandits allowed to have infinitely many arms and proposed OFUL, which has an $\tilde{O}(d\sqrt{T})$ regret bound. However, the previous studies assumed that feature vectors are noiseless, so they cannot be directly applied to our noisy settings.

In our noisy settings, there exists a gap between true mean reward $z_{a,t}^{\top}\theta^{\star}$ and contaminated mean reward $x_{a,t}^{\top}\theta^{\star}$. We discuss some variants of contextual linear bandits for contaminated reward functions. The semi-parametric contextual bandits, where the mean reward for an arm is modeled as a linear function with a bounded confounding term that is equal for all arms, were studied by Greenewald et al. (2017); Krishnamurthy et al. (2018), and Kim and Paik (2019). The algorithm proposed by Krishnamurthy et al. (2018) achieved $\tilde{O}(d\sqrt{T})$ regret in this setting. Another variant is the misspecified setting, where the mean rewards are allowed to have at most $\epsilon\geq 0$ distance from the best-fit linear model. In this setting, Lattimore et al. (2020) achieved an $\tilde{O}(\sqrt{dT}+\epsilon T\sqrt{d})$ regret bound. However, proposed methods for the semi-parametric or misspecified contextual bandits show trivial regret bounds in our setting. This is because the contaminated terms in our setting differ for each arm at each time, and they are unbounded stochastic values.

Some recent studies have considered noise in feature vectors for contextual linear bandits. Lamprier et al. (2018) considered bounded zero-mean feature noise under the assumption that true features for each arm, which are not given to the agent, are fixed over time; they achieved an $\tilde{O}(d\sqrt{T})$ regret bound. Kirschner and Krause (2019) did not fix the true features and considered different noise settings such that the distributions of the contexts were given to the agent each time, but the sampled contexts were hidden. They achieved an $\tilde{O}(d\sqrt{T})$ regret bound. However, we consider that true features are randomly sampled at each time without providing the true feature distribution to the agent, and only randomly sampled noisy features are observed. The noisy features are even allowed to have missing entries, making the problem more challenging.

Several previous studies have considered how to intentionally add noise to contextual linear bandits. Differential private bandit learning (Shariff and Sheffet, 2018; Zheng et al., 2020) added noise to protect privacy. The authors considered adding some noise to the matrices and vectors that contain feature information. Noise vectors are generated from a Gaussian distribution with zero mean and the identity covariance matrix. Importantly, the agent knows the noise distribution to compute an upper confidence bound, which is the main difference from our setting. The algorithms suggested in Shariff and Sheffet (2018) and Zheng et al. (2020) with knowledge of the noise distribution parameters, achieved $\tilde{O}((d+d^{3/4}\epsilon^{-1/2})\sqrt{T})$ for $(\epsilon,\delta)$-differential privacy and $\tilde{O}(T^{3/4}/\epsilon)$ for $(\epsilon,\delta)$-locally differential privacy, respectively. In our noisy feature setting, we consider more complex noise generated from a multivariate Gaussian distribution that allows non-identity covariance matrices with missing entries. Moreover, we consider that the parameters of the noise distribution are not known a priori. Adversarial attacks for linear bandits were also studied by Garcelon et al. (2020), in which an adversary attempts to confuse the agent by adding noise to rewards or features. The authors suggested a way of attacking feature vectors to prevent any algorithm from achieving a sub-linear regret bound, which is the opposite of our purpose.

This paper focuses on i.i.d. feature noise randomly generated from a multivariate Gaussian distribution with missing entries for each arm. This noise cannot be handled by previous approaches, such as the semi-parametric, misspecified, or differential private bandit models, as discussed above. Interestingly, this is the first work to handle missing entries in features for contextual linear bandits as far as we know, even though missing data has been commonly studied for various learning problems such as regression for prediction and PCA (Shang et al., 2014; Ramoni and Sebastiani, 2001; Han, 2014; Bang and Robins, 2005).

Notation. For any $A\in\mathbb{R}^{m\times m}$, we denote by $A^{-1}$ the inverse of $A$ when $A$ is invertible; otherwise, denote by $A^{-1}$ the Moore-Penrose inverse. The $i$-th singular value of $A$ is denoted by $\sigma_{i}(A)$. For any non-negative integer $m$, denote by $I_{m}$ the $m\times m$ identity matrix and by ${\bm{0}_{m\times 1}}$ the $m$-dimensional vector with all one entries. For any sets $B,C\subset\{1,...,m\}$, we denote by $A_{B,C}$ the submatrix of $A$ with row and column indexes in $B,C$, respectively. Then $A_{B,B}^{-1}$ denotes the inverse matrix of $A_{B,B}$. For any $x\in\mathbb{R}^{m}$, $x_{B}$ is a subvector of $x$ with indexes in $B$. For any $x\in\mathbb{R}^{m}$ and $y\in\mathbb{R}^{n}$, denote by $[x;y]\in\mathbb{R}^{m+n}$ the concatenated vector. Lastly, we define $\|x\|_{A}=\sqrt{x^{\top}Ax}$.

2 PROBLEM STATEMENT

Here we describe the contextual linear bandit models with noisy features. Let $\mathcal{A}$ be a set of arms with $|\mathcal{A}|=K$. At each time $t\in[T]$, for all $a\in\mathcal{A}$, the true feature vector $z_{a,t}$ is assumed to be i.i.d. and generated to follow the Gaussian distribution $\mathcal{N}(\nu_{f},\Sigma_{f})$ where $\nu_{f}\in\mathbb{R}^{d}$ and $\Sigma_{f}\in\mathbb{R}^{d\times d}$. The noise vector $\varepsilon_{a,t}$ is also assumed to be i.i.d. and follows the Gaussian distribution $\mathcal{N}({\bm{0}_{d\times 1}},\Sigma_{n})$ where $\Sigma_{n}\in\mathbb{R}^{d\times d}$. The noise feature vector is then defined as $x_{a,t}=(z_{a,t}+\varepsilon_{a,t})\circ m_{a,t}$ where $\circ$ is the element-wise product, $m_{a,t}$ is a masking vector such that $m_{a,t}\in\{0,1\}^{d}$, and each entry in $m_{a,t}$ follows a Bernoulli distribution with parameter $p$. We define an active set of arms $\mathcal{A}_{t}\subseteq\mathcal{A}$ for each time $t$ by removing some outliers in $\mathcal{A}$. We will describe the active set in more detail soon. Then, at each time, an agent observes $\mathcal{A}$ with noisy features $x_{a,t}$ for $a\in\mathcal{A}$ and selects an arm $a_{t}\in\mathcal{A}_{t}$ and receives reward feedback $y_{t}=z_{a_{t},t}^{\top}\theta^{\star}+\eta_{t}$ with a latent parameter $\theta^{\star}\in\mathbb{R}^{d}$ where noise $\eta_{t}$ is i.i.d. and follows a $\sigma$- sub-Gaussian distribution with mean zero. We note that the agent has no a priori knowledge of the noise distribution parameters $\nu_{f},\Sigma_{f},\Sigma_{n},$ and $p$, true feature vectors $z_{a,t}$, and latent model parameter $\theta^{\star}$. The agent can observe noisy features $x_{a,t}$ for all $a\in\mathcal{A}$. For simplicity, we assume that $\|\nu_{f}\|_{2}\leq 1$, $\|\theta^{\star}\|\leq 1$, and $\sigma=1$. In addition, we assume that $\sigma_{1}(\Sigma_{f}+\Sigma_{n})$ and $\sigma_{d}(\Sigma_{f}+\Sigma_{n})$ are strictly positive constants.

The active action set $\mathcal{A}_{t}$ is a subset of $\mathcal{A}$ in which all outliers are removed based on observed features such that

[TABLE]

Since the features follow a Gaussian distribution, the mean reward for an arm may become large, which can induce large regret. Therefore, we devise the active action set $\mathcal{A}_{t}$ to restrict the available regret scale for our theoretical analysis, and show that $\mathcal{A}_{t}$ is the same as $\mathcal{A}$ for all $t$ with high probability, to justify the scaling condition of $\mathcal{A}_{t}$ . Since every $x_{a,t}$ is an i.i.d. Gaussian random variable, from Theorem 1 in Hsu et al. (2012) we can show that for all $a\in\mathcal{A}$ and $t\in[T]$, given $\|m_{a,t}\|_{2}^{2}$, with probability at least $1-1/T$, we have

[TABLE]

which implies that $\mathcal{A}_{t}$ equals $\mathcal{A}$ for all $t\in[T]$ with high probability.

Objective function. The goal of this problem is to design a policy that minimizes regret over a time horizon, which is defined as the difference between the cumulative reward from the optimal policy and the suggested policy. Therefore, it is essential to find the optimal arm at each time. In this problem, we consider a Bayesian perspective for the oracle defined on the observed information, which can be a general framework. For example, in the standard linear bandit where the true features $z_{a,t}$ are observed such that $x_{a,t}=z_{a,t}$, the standard oracle can be interpreted from a Bayesian perspective so that $a_{t}^{\star}\in\arg\max_{a\in\mathcal{A}_{t}}\mathbb{E}[z_{a,t}^{\top}\theta^{\star}|x_{b,t};b\in\mathcal{A}]=\arg\max_{a\in\mathcal{A}_{t}}\mathbb{E}[z_{a,t}^{\top}\theta^{\star}|z_{b,t};b\in\mathcal{A}]=\arg\max_{a\in\mathcal{A}_{t}}z_{a,t}^{\top}\theta^{\star}$. In our noisy setting, the agent can only observe noisy features for each arm. Therefore, determining the Bayesian oracle from noisy observed features is non-trivial. We assume that the oracle has complete knowledge of the latent parameters $\nu_{f},\Sigma_{f},\Sigma_{n},$ and $\theta^{\star}$. Then, given $x_{a,t}$ for $a\in\mathcal{A}$, the oracle’s action at time $t$ from a Bayesian point of view is defined as:

[TABLE]

where the equality comes from the independence of the observed features. We note that $\operatorname*{argmax}_{a\in\mathcal{A}_{t}}z_{a,t}^{\top}\theta^{\star}$ can be too strong for an oracle in our setting, which is not achievable in general to obtain non-trivial regret. This is because $z_{a,t}$ cannot be perfectly recovered from $x_{a,t}$ in general due to unknown independent noise $\varepsilon_{a,t}$ and missing entries for $a\in\mathcal{A}_{t}$ at each time $t$.

Adopting the Bayesian optimal arm $a_{t}^{\star}$, we define regret for a policy of selecting arm $a_{t}$ at time step $t\in[T]$ as follows:

[TABLE]

Challenges. Now, we explain why it is non-trivial to find the Bayesian oracle in our setting even without missing entries where $x_{a,t}=z_{a,t}+\varepsilon_{a,t}$. The true feature is $z_{a,t}=x_{a,t}-\varepsilon_{a,t}$ so the oracle policy is $a^{\star}_{t}\in\arg\max_{a\in\mathcal{A}_{t}}\mathbb{E}\left[z_{a,t}^{\top}\theta^{\star}|x_{a,t}\right]=\arg\max_{a\in\mathcal{A}_{t}}\{x_{a,t}^{\top}\theta^{\star}-\mathbb{E}[\varepsilon_{a,t}^{\top}\theta^{\star}|x_{a,t}]\}.$ Since $\varepsilon_{a,t}$ and $x_{a,t}$ are not independent, we can observe $\mathbb{E}[\varepsilon_{a,t}^{\top}\theta^{\star}|x_{a,t}]\neq 0$ in general, which means that the optimal arm is highly dependent on the distribution of $\varepsilon_{a,t}$ and $z_{a,t}$. Therefore, the Bayesian oracle $a_{t}^{\star}\neq\operatorname*{argmax}_{a\in\mathcal{A}_{t}}x_{a,t}^{\top}\theta^{\star}$ in general, which makes our problem different from standard linear bandits where $x_{a,t}=z_{a,t}$.

To illustrate this, we provide a toy example in Figure 1. We consider Gaussian distributions for $z_{a,t}$ and $\varepsilon_{a,t}$ in two-dimensional space such that $z_{a,t}$ follows a Gaussian distribution with $v_{f}=[0,0]$ and $\Sigma_{f}=[1,0;0,0]$ and $\varepsilon_{a,t}$ follows a Gaussian distribution with mean $[0,0]$ and $\Sigma_{n}=[0,0;0,1]$. We note that $z_{a,t}$ and $\varepsilon_{a,t}$ are independent and orthogonal to each other because $z_{a,t}$ is on the $x$-axis and $\varepsilon_{a,t}$ is on the $y$-axis with probability $1$. We assume that there are two arms, $\{1,2\}$, and the observed features are $x_{1,t}=[1/\sqrt{2},1/\sqrt{2}]$ and $x_{2,t}=[\sqrt{2},-\sqrt{2}]$, respectively. We consider $\theta^{\star}=[1/\sqrt{2},1/\sqrt{2}]$, which points in the same direction as $x_{1,t}$. Then in the standard linear bandits where the observed feature is the true feature ($x_{a,t}=z_{a,t}$), arm $1$ is the optimal arm from the fact that $\mathbb{E}[z_{1,t}^{\top}\theta^{\star}|x_{1,t}]=x_{1,t}^{\top}\theta^{\star}=1$ and $\mathbb{E}[z_{2,t}^{\top}\theta^{\star}|x_{2,t}]=x_{2,t}^{\top}\theta^{\star}=0$. However, this is not the same in our noisy setting where $x_{a,t}=z_{a,t}+\varepsilon_{a,t}$. Since $z_{a,t}$ is on the $x$-axis and $\varepsilon_{a,t}$ is on the $y$-axis, we can estimate $z_{a,t}$ by projecting $x_{a,t}$ on the $x$-axis. Therefore, we obtain $\mathbb{E}[z_{1,t}|x_{1,t}]=[1/\sqrt{2},0]$ and $\mathbb{E}[z_{2,t}|x_{2,t}]=[\sqrt{2},0]$. Then the Bayesian optimal arm is arm $2$ because $\mathbb{E}[z_{1,t}^{\top}\theta^{\star}|x_{1,t}]=1/2$ and $\mathbb{E}[z_{2,t}^{\top}\theta^{\star}|x_{2,t}]=1$. From this example, we can observe that our noisy problem can be totally different from the standard linear bandits where the optimal action is directly determined by observed features. In addition, considering missing entries increases the difficulty of the problem.

As shown in the toy example, we can observe that the optimal arm is highly dependent not only on $x_{a,t}$ and $\theta^{\star}$, but also on the distribution of $z_{a,t}$ and $\varepsilon_{a,t}$. Therefore, the first difficulty is deriving the Bayesian oracle policy. Then, the second difficulty is learning this oracle in a bandit setting where the latent parameters are unknown.

Contribution. First, we analyze the simple case with $p=1$ where there are no missing entries in the observed noisy features. Analyzing the Bayesian oracle strategy, we show that slightly modified OFUL (Abbasi-Yadkori et al., 2011) can achieve an $\tilde{O}(d\sqrt{T})$ regret bound. Then we dive into the more complicated case with missing entries. We show that the Bayesian oracle strategy requires solving non-linear programming in this case. Based on the insight, we design an algorithm that can achieve $\tilde{O}(d\sqrt{T})$ when $K$ is large. Lastly, we examine our algorithm using synthetic and real-world datasets and observe that the results are consistent with our theoretical analysis.

3 WARM-UP: NOISY FEATURES WITHOUT MISSING ENTRIES

In this section, we analyze the simple case of $p=1$ where the observed features have no missing entries but only Gaussian noise. First, we analyze the oracle strategy and then provide an algorithm with regret analysis.

Bayesian oracle strategy. We examine the strategy of a Bayesian oracle that has complete knowledge of latent parameters $\nu_{f}$, $\Sigma_{f}$, $\Sigma_{n}$, and $\theta^{\star}$ in the following proposition.

Proposition 1**.**

Define:

[TABLE]

Then the Bayesian oracle policy under the noisy features without missing entries selects arm $a^{\star}_{t}$ at time step $t\in[T]$ such that:

[TABLE]

Proof.

At each time step $t$, given noisy feature vectors, the Bayesian optimal decision is expressed as:

[TABLE]

From the Bayesian analysis of multivariate Gaussian random variables (Section 3.1 in Smith and Winkler (2006)), we can show that for any $a\in\mathcal{A}_{t}$,

[TABLE]

which concludes the proof. ∎

From Proposition 1, we can observe that $\mathbb{E}[\varepsilon_{a,t}^{\top}\theta^{\star}|x_{a,t}]=x_{a,t}^{\top}\theta^{\star}-\mathbb{E}[z_{a,t}^{\top}\theta^{\star}|x_{a,t}]=x_{a,t}^{\top}\theta^{\star}-[1;x_{a,t}]^{\top}\overline{\theta}\neq 0$ in general, as we discussed in the previous section, which makes the problem different from the standard contextual linear bandits. Although $\mathbb{E}[\varepsilon_{a,t}^{\top}\theta^{\star}|x_{a,t}]\neq 0$, we can observe that interestingly, the noisy contextual bandit without missing entries still models the linear reward with respect to the modified contextual vector $[1;x_{a,t}]$, because the mean reward of an arm $a$ given the observed feature is $\mathbb{E}[z_{a,t}^{\top}\theta^{\star}|x_{a,t}]=[1;x_{a,t}]^{\top}\overline{\theta}$ from $\eqref{eq:bayere}$. Thus, the regret of an algorithm selecting arm $a_{t}$ at time step $t=1,\dots,T$ can be expressed as:

[TABLE]

Algorithm and regret analysis. From (2), the problem can be converted to estimating $\overline{\theta}$ using observed features $[1;x_{a,t}]$ and reward. The observed reward can be considered as Bayesian reward whose mean reward is $[1;x_{a_{t},t}]^{\top}\overline{\theta}$ given $x_{a,t}$’s. From the insight of the Bayesian analysis, we suggest using the strategy OFUL (Abbasi-Yadkori et al., 2011) that was proposed for standard contextual linear bandits. The algorithm is based on the principle of optimism in the face of uncertainty for reward estimation. To build the confidence set to estimate $\overline{\theta}$ in our setting, we consider $\|x_{a,t}\|_{2}=O(\sqrt{d\log(KT)})$ for $a\in\mathcal{A}_{t}$ and $\|\overline{\theta}\|_{2}\leq C_{2}$ for some constant $C_{2}>0$ and tune the bias term $\lambda>0$ to control the variance of the feature vector. Define the confidence set as follows:

[TABLE]

where bias term $\lambda=C_{3}d\log(KT)$, $V_{t}=\lambda I_{d+1}+\sum_{s=1}^{t}[1;x_{a_{t},t}][1;x_{a_{t},t}]^{\top}$, and $\widehat{\theta}_{t}=V_{t}^{-1}\sum_{s=1}^{t}[1;x_{a_{t},t}]y_{t}$ for some constants $C_{1},C_{3}>0$. We provide a pseudocode of the algorithm in Appendix A.1.

The following proposition shows that OFUL can achieve a tight regret bound in the noisy setting without missing entries.

Proposition 2**.**

OFUL* with $\mathcal{C}_{t-1}$ and modified feature vectors $[1;x_{a,t}]$ for all $a\in\mathcal{A}$ and $t\in[T]$ can guarantee $R(T)=\tilde{O}(d\sqrt{T})$.*

Following the proof steps of Theorem 3 in Abbasi-Yadkori et al. (2011) easily reveals the regret bound. We prove Proposition 2 in Appendix A.2.

Tightness of the regret bound. The result in Proposition 2, $\tilde{O}(d\sqrt{T})$, matches the regret lower bound for standard linear bandits, $\Omega(d\sqrt{T})$ (Li et al., 2010), up to poly-logarithmic factors, although our regret is defined based on the Bayesian oracle, unlike previous bandit literature. For the regret lower bound of the standard model, we additionally consider that the norms of true features are bounded by $\tilde{O}(\sqrt{d})$ as in our setting, which scales the regret $\sqrt{d}$ times more compared to the original form in Li et al. (2010). Furthermore, our regret bound is comparable to the results of previous work such as the achieved $\tilde{O}(d\sqrt{T})$ regret bound considering the contaminated mean reward (Krishnamurthy et al., 2018), or $\tilde{O}(d\sqrt{T})$ (Lamprier et al., 2018; Kirschner and Krause, 2019; Shariff and Sheffet, 2018) regret bounds considering feature noise. However, it is an open problem to obtain the problem-specific regret lower bound for our noisy setting under the Bayesian perspective.

4 NOISY FEATURES WITH MISSING ENTRIES

Now, we move on to our main contribution. Here, we consider a case where every component of each observed feature vector has been erased with missing probability $1-p\in(0,1)$. Each component of the feature vector is masked with a random variable following a Bernoulli distribution with parameter $p$. Similar to the previous section, we first derive the Bayesian oracle strategy and suggest an algorithm with regret analysis.

Bayesian oracle strategy. Missing entries affect the information available to the oracle. The Bayesian oracle maximizes the expected reward, knowing the observed feature vectors with missing entries. Given a feature vector $x\in{\mathbb{R}}^{d}$, we define ${\cal S}(x)$ as the indexes for non-missing entries and ${\cal U}(x)$ as the indexes for missing entries. For simplicity, we use $\mathcal{S}$ for $\mathcal{S}(x)$ and $\mathcal{U}$ for $\mathcal{U}(x)$ if there is no confusion. Then, the Bayesian oracle strategy arises from the following theorem.

Theorem 1**.**

For any $x\in{\mathbb{R}}^{d},\nu_{f}\in{\mathbb{R}}^{d},$ and $\Sigma\in{\mathbb{R}}^{d\times d}$, define $\overline{x}(\nu,\Sigma,x)\in\mathbb{R}^{d}$ as:

[TABLE]

Then the Bayesian oracle strategy under the noisy features with missing entries selects arm $a^{\star}_{t}$ at time step $t\in[T]$ such that:

[TABLE]

We note that when $S=[d]$, we observe $\overline{x}(\nu_{f},\Sigma_{f}+\Sigma_{n},x_{a,t})=x_{a,t}$, which is the same as in the case of noisy features without missing entries. When $S=\emptyset$, we observe $\overline{x}(\nu_{f},\Sigma_{f}+\Sigma_{n},x_{a,t})=\nu_{f}$, which implies that the oracle estimates true features from the mean value directly because the oracle does not have noisy feature information.

Proof.

Denote by $x^{\prime}_{a,t}$ the noisy feature vector for arm $a$ without masking such that $x^{\prime}_{a,t}=z_{a,t}+\varepsilon_{a,t}$. Then we observe that $x_{a,t}^{\prime}$ follows the Gaussian distribution with mean $\nu_{f}$ and covariance matrix $\Sigma_{f}+\Sigma_{n}$. Then by using the conditional Gaussian distribution from Proposition 3.13 in Eaton (1983), we can show that for all $a\in\mathcal{A}_{t}$,

[TABLE]

which is the estimated feature vector for recovering missing values. Thus, from (2), (5), and $\overline{\theta}=[\nu^{\top}\theta^{\star}-\nu^{\top}\theta^{\prime};\theta^{\prime}]$, the expected reward given the observed feature vector can be expressed as

[TABLE]

which concludes the proof. ∎

From Theorem 1, the Bayesian oracle strategy selects an arm based on $\overline{x}(\nu_{f},\Sigma_{f}+\Sigma_{n},x_{a,t})$, which implies that the model is non-linear with respect to the observed feature $x_{a,t}$. This is a significant difference from the simple case of noisy features without missing entries. From the oracle strategy in Theorem 1 and (10), we can show that regret can be expressed as:

[TABLE]

Algorithms and regret analysis. From Theorem 1, the oracle policy is not linear to the observed feature vectors, which implies that we cannot naively use OFUL to guarantee a sub-linear regret bound when the distribution latent parameters are not given to the algorithm. Therefore, we propose Algorithm 1, which includes a procedure to estimate Bayesian features $[1;\overline{x}(\nu_{f},\Sigma_{f}+\Sigma_{n},x_{a,t})]$.

Algorithm 1 estimates the parameters of the distribution for the feature vectors from the information of the observed features at each time. Using the estimated distribution parameters, the algorithm estimates the Bayesian feature $[1;\overline{x}(\nu_{f},\Sigma_{f}+\Sigma_{n},x_{a,t})]$ as $\widehat{z}_{a,t}$ for all $a\in\mathcal{A}_{t}$. Furthermore, the algorithm rarely updates the estimated features for previously chosen arms using the current estimated distribution parameters while reducing the computation cost. Finally, it takes a strategy based on the principle of optimism in the face of uncertainty using estimated feature vectors $\widehat{z}_{a,t}$. The algorithm selects $a_{t}$ and observes reward $y_{t}$ for each time $t$. We note that the algorithm considers the observed reward as Bayesian reward whose mean reward is $[1;\overline{x}(\nu_{f},\Sigma_{f}+\Sigma_{n},x_{a_{t},t})]^{\top}\overline{\theta}$ from (10).

The necessary notations for Algorithm 1 are defined as follows. First we define $\lambda=C_{4}d\log(KT)$, $V_{t}=\lambda I_{d+1}+\sum_{s=1}^{t}\widehat{z}_{a_{s},s}\widehat{z}_{a_{s},s}^{\top}$, $\widehat{\theta}_{t}=V_{t}^{-1}\sum_{s=1}^{t}\widehat{z}_{a_{s},s}y_{s}$, and $X_{t}=\sum_{s=1}^{t}\|\widehat{z}_{a_{s},s}\|_{V_{t}^{-1}}$ for some large constant $C_{4}>0$. Note that $\|\overline{\theta}\|_{2}\leq C_{2}$ for some constant $C_{2}>0$ and that the bias term $\lambda$ is tuned to control the variance of the observed feature norms. In addition, note that $\widehat{p}$ is an estimator for $p$ at each time in the algorithm. Then, we define the confidence set for estimating a latent parameter $\overline{\theta}$ as

[TABLE]

for some large constants $C_{5},C_{6}>0$. The estimation error of Bayesian features incurs that Algorithm 1 requires an additional confidence bound term, $C_{6}(d/\widehat{p})^{3/2}\sqrt{\log(KT)/K}X_{t}$, in (12) compared to OFUL. By solving the convex problem to find $a_{t}$ in Algorithm 1, as stated in Section 19.3 in Lattimore and Szepesvári (2020), the action in each round can be simply calculated as

[TABLE]

Then, the following theorem provides regret for the algorithm.

Theorem 2**.**

Algorithm 1 achieves the regret bound of

[TABLE]

The first term for the regret bound in Theorem 2 comes from the analysis of optimism in the face of uncertainty, and the second and third terms come from the estimation error for Bayesian features, which is the main difference from standard linear bandits where the true feature vectors are given. We can note that the second term dominates over the third one when $T$ is large. An additional factor $d$ compared to Proposition 2 comes from the estimation error of Bayesian features, and factor $1/p^{3/2}$ mainly comes from estimation error of $\widehat{\Sigma}$. From Theorem 2, when $K=\Omega(\max\{d^{2}/p^{3},1/(\sqrt{T}p^{4})\})$ we get

[TABLE]

Therefore, the regret becomes insensitive to $p$ when the algorithm has sufficient observed feature information to estimate Bayesian features accurately. We provide a further discussion about the tightness of the bound with respect to $p$ in Appendix A.4. We also note that by using the well-known doubling trick, we can run our algorithm without information of $T$ a priori.

Proof sketch.

We provide a proof sketch here; Appendix A.3 contains the full version. Let $\overline{z}_{a,t}=[1;\overline{x}(\nu,\Sigma_{f}+\Sigma_{n},x_{a,t})]$ for simplicity, and instantaneous regret $r_{t}=\overline{z}_{a^{\star}_{t},t}^{\top}\overline{\theta}-\overline{z}_{a_{t},t}^{\top}\overline{\theta}$. For $s\leq t$, we write $\widehat{\nu}_{t}$, $\widehat{\Sigma}_{t}$, $\widehat{z}_{a,s}(t)$, and $\widehat{p}_{t}$ for $\widehat{\nu}$, $\widehat{\Sigma}$, $\widehat{z}_{a,s}$, and $\widehat{p}$ at time step $t$, respectively. From the condition for the active set, we have $\|\overline{z}_{a,t}\|_{2}=O(\sqrt{d\log(KT)})$ for all $a\in\mathcal{A}_{t}$. The confidence set $\mathcal{C}_{t}$ is designed on this constraint for $\overline{z}_{a,t}$. However, the algorithm selects an arm based on estimated features $\widehat{z}_{a,t}(t)$ rather than $\overline{z}_{a,t}$. Hence, the first few steps require collecting feature information to estimate $\overline{z}_{a_{s},s}$ as $\widehat{z}_{a_{s},s}(t)$ to satisfy $\|\widehat{z}_{a_{s},s}(t)\|_{2}=O(\sqrt{d\log(KT)})$. Let $\tau=2\lceil d(\log(T))^{2}/(Kp^{4})\rceil$. Then, we show that for all $t>\tau$ and $1\leq s\leq t$, with high probability we have

[TABLE]

To obtain a regret bound before $\tau$, we use the fact that $x_{a}$ follows a Gaussian distribution given $m_{a,t}$. Using a bound for an expected maximum Gaussian variable, we can show that

[TABLE]

Then from (14), the regret bound for the first $\tau$ time steps can be obtained as

[TABLE]

Let $\tilde{\theta}_{t}=\arg\max_{\theta\in\mathcal{C}_{t-1}}\max_{a\in\mathcal{A}_{t}}\langle\widehat{z}_{a,t}(t),\theta\rangle$. Now, we analyze regret after the time step $\tau$. Our algorithm selects an arm $a_{t}$ based on the estimated features $\widehat{z}_{a,t}(t)$ and observes $y_{t}$ whose mean reward is $\overline{z}_{a_{t},t}^{\top}\overline{\theta}$ from the Bayesian view. Therefore, regret is influenced by both estimation errors for $\tilde{\theta}_{t}$ and $\widehat{z}_{a,t}(t)$, which is the main difference from the standard linear bandits in regret analysis. First, we try to separate regret by the estimation errors; we show that with high probability,

[TABLE]

where the instantaneous regret bound comprises the estimation errors for $\widehat{z}_{a_{t}^{\star},t}$, $\widehat{z}_{a_{t},t}(t)$, and $\tilde{\theta}_{t}$. To obtain the error bounds for $\|\overline{z}_{a,t}-\widehat{z}_{a,t}(t)\|_{2}$ and $\|\overline{\theta}-\tilde{\theta}_{t}\|_{2},$ we consider the estimation errors for $\widehat{\nu}_{t}$ and $\widehat{\Sigma}_{t}$. Using vector and matrix concentration inequalities, we can show that with high probability

[TABLE]

From (13), (17), (18), and since $\widehat{p}_{t}=\Theta(p)$ with high probability from Hoeffding’s inequality, we can show that $\overline{\theta}\in\mathcal{C}_{t-1}$ with high probability. In addition, we show that $X_{t-1}=O\left(\sqrt{td\log T}\right).$ Then, with high probability, we have

[TABLE]

From (17), (18), and (20), we have

[TABLE]

Using (13) and Lemma 11 in Abbasi-Yadkori et al. (2011), we can show that with high probability

[TABLE]

Finally, with (15), (22), and (23), using the Cauchy-Schwarz inequality, we have

[TABLE]

which concludes the proof. ∎

Algorithm efficiency. Algorithm 1 rarely updates the estimated features for previously chosen arms, achieving the $O(T\log(T))$ computation cost over the horizon $T$. The update procedure requires $O(t)$ memory space for each time step $t$. This update is required to obtain well-estimated features for the previously chosen arms, which are then used to obtain $V_{t}$, $\widehat{\theta}_{t}$, and $X_{t}$. We propose a more efficient algorithm (Algorithm 3 in Appendix A.5) to reduce the computation cost and the required memory space for Algorithm 1, achieving the $O(T)$ computation cost and $O(1)$ memory space. However, the algorithm requires information of $\alpha>0$ that satisfies $\alpha\leq p$. We can show that this algorithm achieves $R(T)=\tilde{O}\left(d\sqrt{T}+(d^{2}/p^{3/2})\sqrt{T/K}+d/(\alpha^{4}K)\right).$ Appendix A.5 contains a more detailed explanation and regret analysis of the algorithm.

5 NUMERICAL EXPERIMENTS

In this section, we present numerical results of experiments for synthetic and real-world datasets.111Our code is available at https://github.com/junghunkim7786/contextual_linear_bandits_under_noisy_features We repeat each experiment $10$ times. First, we describe the experimental setting for synthetic datasets. For the latent parameters, we generate $\theta^{\star}$ and $\nu_{f}$ from the uniform distribution in $[0,1]$ and normalize them using $2$-norm. Then, we generate random matrices $A,B\in[0,1]^{d\times d}$ from the uniform distribution and construct the covariance matrices as $\Sigma_{f}=A^{\top}A$ and $\Sigma_{n}=B^{\top}B$ with spectral norm normalization. Feature vectors are generated from $\mathcal{N}(\nu_{f},\Sigma_{f})$, noise vectors are generated from $\mathcal{N}(0,\Sigma_{n})$, and missing entries are assigned following a Bernoulli distribution with probability $p$. Using the distributions, we generate $K$ number of noisy feature vectors for each time. The reward noise $\eta_{t}$ is generated from $\mathcal{N}(0,1)$. We set $d=2$ and $T=10^{4}$.

For real-world experiments, we use Avazu click-through rate (CTR) dataset (Avazu, 2014) that contains ‘clicked or not’ information for each advertisement recommendation. We use an autoencoder model to preprocess ad-user pair feature information for constructing item feature vectors. We set preprocessed feature dimension $d=32$, $K=20$, and $T=10^{5}$. We erase some entries in the preprocessed feature vectors with missing probability $1-p=0.1$. A more detailed explanation is deferred to Appendix A.6.

We compare the performance of Algorithm 1 with OFUL described in Section 3. We first examine cumulative regret over the time horizon $T$ for each algorithm in synthetic datasets varying $1-p$ from [math] to $0.4$, where the number of missing entries is likely to increase as the missing probability increases. Figure 2 (a,b) confirms that our algorithm performs better than OFUL for various $1-p$. In Figure 2 (a) when $K=30$, the regret of our algorithm increases rapidly as the missing probability increases, while in Figure 2 (b) when $K=100$, we can observe that our algorithm becomes more robust to the missing probability. This result is consistent with Theorem 2, which shows that our algorithm is insensitive to $p$ with $R(T)=\tilde{O}(d\sqrt{T})$ when $K$ is sufficiently large. Furthermore, we can observe that the regret variance for Algorithm 1 is smaller than that for OFUL.

Next, in Figure 2 (c), we examine the cumulative regret over time steps $t\in[T]$ when $1-p=0.3$ and $K=100$ in synthetic datasets. We can observe that Algorithm 1 achieves much better performance with sub-linearly increasing regret with low variance, whereas OFUL exhibits almost linearly increasing regret with large variance. In Figure 2 (d), we demonstrate that our algorithm achieves higher CTRs than OFUL in the real-world dataset. Further experiments for various real-world datasets are provided in Appendix A.6.

6 DISCUSSION ON EXTENSIONS

In our setting, we consider that the features of the arms are generated from the same Gaussian distribution. However, our method can be easily extended to the case where the features of the arms are generated from different Gaussian distributions with each other. In such a case, the algorithm needs to estimate latent distribution parameters for each arm. Then, with a slight modification to our algorithm, we can achieve $R(T)=\tilde{O}((d^{2}/p^{3/2})\sqrt{T}+d/p^{4})$. In addition, in our setting, we consider that the features of arms at the same time are independent. We can extend our setting to the dependent case such that the $K$ number of features is generated from a Gaussian distribution with parameters $\nu_{f}\in\mathbb{R}^{Kd}$ and $\Sigma_{f}\in\mathbb{R}^{Kd\times Kd}$ having a Gaussian noise with $\Sigma_{n}\in\mathbb{R}^{Kd\times Kd}$. With a slight modification of our algorithm, we can achieve $R(T)=\tilde{O}(((Kd)^{2}/p^{3/2})\sqrt{T}+(dK)/p^{4})$.

In our setting, we consider the Gaussian conjugate distribution of features for Bayesian analysis. It would be of interest to extend our setting to other conjugate distributions.

7 CONCLUSION

In this paper, we studied contextual linear bandits under noisy features that contain Gaussian noise and missing entries. We analyzed Bayesian oracles and subsequently proposed an algorithm to achieve $\tilde{O}(d\sqrt{T})$ when the number of arms is large. Lastly, we demonstrated the performance of our algorithm using synthetic and real-world datasets.

Limitation: We leave several open questions. For instance, extensions to other distributions beyond the Gaussian distribution for Bayesian analysis would be of interest. In addition, obtaining a regret lower bound would be useful in understanding the fundamental limitations of our model.

Potential negative societal impacts: This study focuses on theoretical analysis; therefore, we could not see any negative social consequences.

Acknowledgements

This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grants funded by the Korea government(MSIT) (No.2022-0-00311, Development of Goal-Oriented Reinforcement Learning Techniques for Contact-Rich Robotic Manipulation of Everyday Objects; No.2019-0-00075, Artificial Intelligence Graduate School Program(KAIST)). The authors would like to thank the anonymous reviewers for their helpful comments and suggestions.

Appendix A APPENDIX

A.1 Pseudocode of OFUL

A.2 Proof of Proposition 2

From OFUL in Abbasi-Yadkori et al. (2011), the algorithm selects an arm $a_{t}$ at each time $t$ such that

[TABLE]

For the completeness, we provide a proof following the proof steps of Theorem 3 in Abbasi-Yadkori et al. (2011). We first provide a lemma about a property of the confidence set $\mathcal{C}_{t-1}$. From Section 3.1 in Smith and Winkler (2006), we observe that $y_{a_{t}}$ given $x_{a,t}$’s follows a $C$-subGuassian distribution with mean $[1;x_{a,t}]^{\top}\overline{\theta}$ for some constant $C>0$.

Lemma 1** (Theorem 2 in Abbasi-Yadkori et al. (2011)).**

With probability at least $1-1/T$, for all $t\geq 0$, $\overline{\theta}$ lies in the set $\mathcal{C}_{t-1}.$

Let instantaneous regret $r_{t}=[1;x_{a^{\star}_{t},t}]^{\top}\overline{\theta}-[1;x_{a_{t},t}]^{\top}\overline{\theta}$. Then from Lemma 1, with probability at least $1-1/T$, for all $t\in[T]$ we have

[TABLE]

where the first inequality comes from the strategy of the algorithm for selecting $a_{t}$ under $\overline{\theta}\in C_{t-1}$.

From Lemma 1, we can obtain that with probability at least $1-1/T$, for all $t\in[T]$,

[TABLE]

where $\lambda=C_{3}d\log(KT)$ for some constants $C_{1},C_{2},C_{3}>0$. We denote by $E$ the event of (27). From (26), (27), and the fact that $r_{t}=O(\sqrt{d\log(KT)})$ from the scale condition for $\mathcal{A}_{t}$, we can obtain

[TABLE]

In what follows, we provide a lemma for a bound of $\sum_{t=1}^{T}\|[1;x_{a_{t},t}]\|_{V_{t-1}^{-1}}$.

Lemma 2**.**

We have

[TABLE]

Proof.

From the scale condition of $\|x_{a,t}\|_{2}$ for $a\in\mathcal{A}_{t}$, there exists a constant $C>0$ such that

[TABLE]

Then for sufficiently large $C_{3}>0$ with $\lambda=C_{3}d\log(KT)$, we have

[TABLE]

From (29), by following the proof steps in Lemma 11 in Abbasi-Yadkori et al. (2011), we have

[TABLE]

∎

Finally from (28) and Lemma 2, we can obtain

[TABLE]

Further discussion. OFUL achieved $\tilde{O}(d\sqrt{T})$ Abbasi-Yadkori et al. (2011) in the standard linear bandits, with mean reward scale bounded by $1$. Therefore, it may be of interest that OFUL, in our setting, still achieves $\tilde{O}(d\sqrt{T})$ for the case of mean reward scale $|[1;x_{a,t}]^{\top}\overline{\theta}|=\tilde{O}(\sqrt{d})$ with $\|[1;x_{a,t}]\|_{2}=\tilde{O}(\sqrt{d})$ and $\|\overline{\theta}\|_{2}=O(1)$. This can be obtained from that we tune a bias term $\lambda$ properly for dealing with the variance of $\|[1;x_{a_{t},t}]\|_{V_{t-1}^{-1}}^{2}$ shown in (29).

A.3 Proof of Theorem 2

For the notational simplicity, let $\Sigma=\Sigma_{f}+\Sigma_{n}$ and $\overline{z}_{a,t}=[1;\overline{x}(\nu,\Sigma,x_{a,t})]$. Also, for $s\leq t$, we write $\widehat{z}_{a,s}(t)$ for $\widehat{z}_{a,s}$ at time step $t$. From the condition for the active set, we observe that $\|\overline{z}_{a,t}\|_{2}=O(\sqrt{d\log(KT)})$. The confidence set $\mathcal{C}_{t}$ is designed considering this constraint to get a tight bound. However, the algorithm selects an arm based on estimated features $\widehat{z}_{a,t}(t)$ instead of $\overline{z}_{a,t}$. Hence, for the first few steps, it requires to collect feature information to estimate $\widehat{z}_{a_{s},s}(t)$ for satisfying $\|\widehat{z}_{a_{s},s}(t)\|_{2}=O(\sqrt{d\log(KT)})$. Recall that $\theta^{\prime}=(\Sigma_{f}+\Sigma_{n})^{-1}\Sigma_{f}\theta^{\star}$ and $\overline{\theta}=[\nu^{\top}\theta^{\star}-\nu^{\top}\theta^{\prime};\theta^{\prime}].$ Let $\tau=2\lceil d(\log(T))^{2}/Kp^{4}\rceil$. Then for any $t>\tau$ and $0<s\leq t$ we have $\|\widehat{z}_{a_{s},s}(t)\|_{2}=O(\sqrt{d\log(KT)})$ with a high probability, which will be shown later. The regret of Algorithm 1 can be decomposed into the followings:

[TABLE]

We use the following lemma to get a regret bound before $\tau$ time steps.

Lemma 3**.**

For any time $t>0$, we have

[TABLE]

Proof.

For any given $m_{a,t}$, $\overline{x}(\nu_{f},\Sigma,x_{a,t})^{\top}\theta^{\prime}$ is a random variable with a Gaussian distribution which mean is $\nu_{f}^{\top}\theta^{\prime}$ for all $a\in\mathcal{A}$. For analyzing the random variable, we provide a bound for the expected maximum value of random variables according to an i.i.d Gaussian distribution in the following.

Lemma 4**.**

Let $X_{i}$ be an independent random variable with $\mathcal{N}(\mu,\sigma_{i}^{2})$ for all $i\in\{1,...,n\}$. Define $X^{\star}=\max\limits_{i\in\{1,\dots,n\}}X_{i}$ and $\sigma^{\star}=\max\limits_{i\in\{1,\dots,n\}}\sigma_{i}$. Then,

[TABLE]

Proof.

For any $\lambda\geq 0$,

[TABLE]

Set $\lambda=\frac{\sqrt{2\log(n)}}{\sigma^{\star}},$ then we get

[TABLE]

∎

Denote by $Var(X)$ the variance of a random variable $X$ and $V^{\star}$ the maximum value among $Var(\overline{x}(\nu_{f},\Sigma,x_{a,t})^{\top}\theta^{\prime})$ for all $a\in\mathcal{A}$. Then using Lemma 4 and $\mathcal{A}_{t}\subseteq\mathcal{A}$, we have

[TABLE]

where $V^{\star}$ is bounded as follows. Define $\mathcal{E}=\{1,...,d\}$. For any $a\in\mathcal{A}$ and any given $m_{a,t}$, we have

[TABLE]

∎

Therefore from Lemma 3, the regret over the first $\tau$ time steps in (LABEL:eq:overRT) is replaced by

[TABLE]

Now we provide a regret bound after $\tau$ time steps. Recall that instantaneous regret $r_{t}=\overline{z}_{a^{\star}_{t},t}^{\top}\overline{\theta}-\overline{z}_{a_{t},t}^{\top}\overline{\theta}$. In the algorithm, at each time step, the covariance matrix $\Sigma$ is estimated by the scaled empirical covariance matrix $\widehat{\Sigma}$ and the mean of feature vector $\nu_{f}$ is estimated by $\widehat{\nu}$. For ease of presentation, we write $\widehat{p}_{t}$, $\widehat{\nu}_{t}$, and $\widehat{\Sigma}_{t}$ for $\widehat{p}$, $\widehat{\nu}$, and $\widehat{\Sigma}$ at time step $t$, respectively. For analyzing the regret, we can decompose the regret according to the estimation errors with high probability as follows:

[TABLE]

which will be shown later. The instantaneous regret bound consists of the estimation errors of $\widehat{z}_{a_{t}^{\star},t}(t)$, $\widehat{z}_{a_{t},t}(t)$, and $\tilde{\theta}_{t}$. Therefore we first focus on providing bounds for $\|\overline{z}_{a,t}-\widehat{z}_{a,t}(t)\|_{2}$ and $\|\overline{\theta}-\tilde{\theta}_{t}\|_{2}.$ From matrix concentration inequalities, we have the following lemma.

Lemma 5**.**

For all $t>\tau/2$, with probability at least $1-3/T$, we have

[TABLE]

Proof.

Recall $x^{\prime}_{a,t}=z_{a,t}+\varepsilon_{a,t}$. We define $v_{a,t}=x^{\prime}_{a,t}\circ m_{a,t}-\nu_{f}p$ for all $a\in\mathcal{A}$. Let $z_{j}$ be the $j$-th entry in an arbitrary vector $z$. Then, for any $u\in\mathbb{R}^{d}$ such that $\|u\|_{2}=1$ and for any $\lambda\in\mathbb{R}$, we get

[TABLE]

where the first inequality is from the normal moment generating function and the third inequality is from Hoeffding’s lemma. From the definition of sub-gaussian vector, we can find that $v_{a,t}$ for $a\in\mathcal{A}$ and $t>0$ are according to independent sub-gaussian with variance proxy $\|\Sigma\|_{2}+1$. According to Theorem 2.1 of Hsu et al. (2012), for all $k>0$ we get

[TABLE]

Then we have with probability at least $1-1/T^{2}$

[TABLE]

By Hoeffding’s inequality, we also have

[TABLE]

Recall that $\tau/2=\lceil d(\log(T))^{2}/(Kp^{4})\rceil$. Then for $t>\tau/2$, we have $\widehat{p}_{t}=\Theta(p)$ at least probability $1-1/T^{2}$. Using the above inequalities, we provide bounds for $\|\nu_{f}-\widehat{\nu}_{t}\|_{2}$ and $\|\Sigma-\widehat{\Sigma}_{t}\|_{2}$. With probability at least $1-1/T^{2}$, for $t>\tau/2$ we have

[TABLE]

Now we provide a bound for $\|\Sigma-\widehat{\Sigma}_{t}\|_{2}$. We define $w=\nu_{f}p$. By using $x_{a,t}=v_{a,t}+w$, we get

[TABLE]

For the first term in the RHS of (43), from Proposition 2.1 of Vershynin (2012) with sub-Gaussian $v_{a}$, with probability at least $1-1/T^{2}$ we get

[TABLE]

For the second term in the RHS of (43),

[TABLE]

From the results of (45) and (41), with probability at least $1-1/T^{2}$ we get

[TABLE]

By putting the results of (43), (44), and (47), with probability at least $1-1/T^{2}$ we get

[TABLE]

We write $Z_{t}$ for $Z$ at time step $t$ in the algorithm. Then, (48) directly implies that

[TABLE]

Let $\textbf{1}_{d\times d}$ be a matrix $\in\mathbb{R}^{d\times d}$ with all $1$ entries and $P=(p-p^{2})I_{d}+p^{2}{\bm{1}_{d\times d}}$. Then from Proposition 1 in Pavez and Ortega (2020), we can show that

[TABLE]

Lastly, with probability at least $1-1/T^{2}$, we get

[TABLE]

Therefore, using the union bound for all time $t>\tau/2$, we can conclude the proof. ∎

From Lemma 5, we define an event

[TABLE]

which holds true with at least probability $1-3/T$. Using Weyl’s inequality, we can show that $\sigma_{d}(\widehat{\Sigma}_{t})\geq\sigma_{d}(\Sigma)-\|\widehat{\Sigma}_{t}-\Sigma\|_{2}$. Under $E_{1}$, for all $t>\tau/2$, we have $\|\Sigma-\widehat{\Sigma}_{t}\|_{2}=o(1).$ Then, we have

[TABLE]

By following the same steps, for any non empty set $\mathcal{S}$, we can also show that

[TABLE]

Then, under $E_{1}$, from the definition of $\widehat{z}_{a,t}(t)$ and (54), for any $t>\tau$ and $0<s\leq t$ we have

[TABLE]

This is because the algorithm updates the estimated features of previously chosen arms once within $\tau/2<t\leq\tau$. In what follows, we provide a bound for feature estimators using (54). For simplicity, we define $\overline{z}_{a,t}\coloneqq[1;\overline{x}(\nu,\Sigma,x_{a,t})]$. In the algorithm, recall that for $s\leq t$, $\widehat{z}_{a,s}(t)=[1;\overline{x}(\widehat{\nu},\widehat{\Sigma},x_{a,s})]$ at time $t$.

Lemma 6**.**

Under $E_{1}$, for all $t>\tau/2$ and $a\in\mathcal{A}_{t}$, we have

[TABLE]

Proof.

First, at time step $t$, we have

[TABLE]

Thus with (59), (54), and (55), under $E_{1}$ for $t>\tau/2$

[TABLE]

Therefore, for any $t>\tau/2$ and $a\in\mathcal{A}_{t},$ it follows that under $E_{1}$,

[TABLE]

∎

Now, we provide a lemma for showing a good property of the confidence set $\mathcal{C}_{t}$.

Lemma 7**.**

Under $E_{1}$, for all $t>\tau$, with probability at least $1-1/T$, we have $\overline{\theta}\in\mathcal{C}_{t-1}.$

Proof.

Since, under $E_{1}$, $\|\widehat{\Sigma}\|_{2}=O(\|\Sigma\|_{2})$ and $\|\widehat{\Sigma}^{-1}\|_{2}=O(\|\Sigma^{-1}\|_{2})$, we have, for all $t>\tau$ and $1\leq s\leq t$ considering rarely updating estimators in the algorithm,

[TABLE]

This is because the algorithm updates the estimated features of previously chosen arms once within $\tau/2<t\leq\tau$. Then for $t>\tau$ we can observe that $\|\widehat{z}_{a_{s},s}(t)\|_{2}^{2}\leq C_{1}d\log(KT)$ and $\|\overline{\theta}\|_{2}\leq C_{2}$ for some constants $C_{1}$ and $C_{2}>0$. We also observe that noise of reward $\eta_{t}$ is independent to $E_{1}$. Using the facts, Lemma 6, and Theorem 2 in Abbasi-Yadkori et al. (2011), we can easily prove that for all $t>\tau$, with probability at least $1-1/T$, $\overline{\theta}$ lies in $\mathcal{C}_{t-1}$. In what follows, we provide details. From Section 3.1 in Smith and Winkler (2006), Proposition 3.13 in Eaton (1983), and $1$-subGaussian $\eta_{t}$, we observe that $y_{a_{t}}$ given $x_{a,t}$ follows a $C_{3}$-subGaussian distribution with mean $\overline{x}^{\top}\overline{\theta}$ for some constant $C_{3}>0$. We denote by $\tilde{\eta}_{t}$ the noise term for $y_{a_{t}}$ given $x_{a,t}$ at time $t$. Then, we have

[TABLE]

Then $\forall x\in\mathbb{R}^{d+1}$, we have

[TABLE]

By setting $x=V_{t}(\widehat{\theta}_{t}-\overline{\theta})$, we have

[TABLE]

From Lemma 9 in Abbasi-Yadkori et al. (2011), with probability at least $1-1/T$ we have

[TABLE]

We can observe that $\widehat{p}_{t}=\Theta(p)$ when $t>\tau$ under $E_{1}$. Putting them together, for some constant $C_{5},C_{6}>0$, with probability at least $1-1/T$, we have

[TABLE]

∎

From the above lemma, we define an event

[TABLE]

Then we provide a lemma for decomposing the instantaneous regret according to the estimation errors of $\widehat{z}_{a_{t}^{\star},t}(t)$, $\widehat{z}_{a_{t},t}(t)$, and $\tilde{\theta}_{t}$.

Lemma 8**.**

Under $E_{2}$, we have

[TABLE]

Proof.

Under $E_{2}$, from the fact that $\overline{\theta}\in\mathcal{C}_{t-1}$ and considering that $a_{t}$ is the chosen arm at time $t$, we have $\widehat{z}_{a_{t}^{\star},t}(t)^{\top}\overline{\theta}\leq\widehat{z}_{a_{t},t}(t)^{\top}\tilde{\theta}_{t}$. Then we have

[TABLE]

∎

Then, we provide a bound for $\sum_{t=\tau+1}^{T}\mathbb{E}[r_{t}]$ in the following.

Lemma 9**.**

We have

[TABLE]

Proof.

Under $E_{1}$ and $E_{2}$, we have $\tilde{\theta}_{t}\in\mathcal{C}_{t-1}$. Then we get

[TABLE]

where the last equality comes from Lemma 11 in Abbasi-Yadkori et al. (2011).

Then with Lemma 8, we have

[TABLE]

where the equality comes from $\tilde{\theta}_{t}\in\mathcal{C}_{t-1}$. Then from Lemma 6, we have

[TABLE]

We note that under the complement event $E_{1}^{c}\cup E_{2}^{c}$ which holds at most probability $2/T$, for all $t>\tau$, we have $r_{t}\leq|\overline{z}_{a^{\star}_{t},t}^{\top}\theta^{\prime}|+|\overline{z}_{a_{t},t}^{\top}\theta^{\prime}|=O(\sqrt{d\log(KT)}).$ Therefore with (LABEL:eq:r_bd), we have

[TABLE]

∎

Now we provide a lemma to bound the first term in (60).

Lemma 10**.**

Under $E_{1}$, we have

[TABLE]

Proof.

Under $E_{1}$, for some sufficiently large constant $C>0$ with $\lambda=Cd\log(KT)$, for any $t\geq\tau+1$ and $s\leq t$ we have

[TABLE]

[TABLE]

Let $2^{l}$ be the smallest time step after $\tau+1$ for an integer $l$ and $2^{n}$ be the largest time step before $T$ for an integer $n$. Then we have $n=O(\log T)$. We note that estimated features of previously selected arms in $V_{t-1}$ are not updated during the time steps between $t=2^{i}+1$ and $2^{i+1}-1$ for $i\in[n]$. Then by following the proof steps in Lemma 11 in Abbasi-Yadkori et al. (2011) and considering rarely updating procedure in the algorithm, we can obtain

[TABLE]

where the first inequality is obtained from $x\leq 2\log(1+x)$ when $x\in[0,1]$, the second and last inequality is obtained from Lemma 11 in Abbasi-Yadkori et al. (2011), and the last equality is obtained from $n=O(\log T)$. ∎

Finally using (LABEL:eq:overRT), (40), Lemmas 9 and 10, and Cauchy-Schwarz inequality, we get

[TABLE]

A.4 Tightness of the regret bound of Theorem 2 with respect to $p$

Dependency on $p$ in regret analysis comes from variance of estimator $\widehat{\Sigma}_{t}$ which includes $1/\widehat{p}_{t}^{2}$ term for reducing bias. In more detail, from (17) we can observe that regret depends on the estimation error for Bayesian feature estimator $\widehat{z}_{a}$, and this error depends on the variance of $\widehat{\Sigma}_{t}$ and $\|x_{a,t}\|_{2}$ (See proof of Lemma 6). Then, from Lemma 5, we get $\|\widehat{\Sigma}_{t}-\Sigma\|_{2}=O((1/p^{2})\sqrt{d\log(T)/tK})$, and from Gaussian distribution we get $\|x_{a,t}\|_{2}=\tilde{O}(\sqrt{pd})$. Therefore, $1/p^{3/2}$ in the regret bound seems to be inevitable considering $1/p^{2}$ in the variance of $\widehat{\Sigma}_{t}$ and $\sqrt{p}$ from $\|x_{a,t}\|_{2}$.

Next, we explain why $1/p^{4}$ appears in the regret bound. We can observe that the instantaneous regret depends on $\|\widehat{z}_{a_{t}}\|_{2}$ from (22) and Lemma 10. For bounding the variance $\|\widehat{z}_{a_{t}}\|_{2}$, we need to bound $\|\widehat{\Sigma}^{-1}_{t}\|_{2}$ from the Bayesian feature definition. Using Weyl’s inequality, we have $\|\widehat{\Sigma}^{-1}_{t}\|_{2}\leq 1/(\sigma_{d}(\Sigma)-\|\widehat{\Sigma}_{t}-\Sigma\|_{2})$ in (54). Therefore, for getting a tight constant bound for $\|\widehat{\Sigma}^{-1}_{t}\|_{2}$, we need at least $t=\Theta(1/p^{4})$ time steps to collect enough information for reducing $\|\widehat{\Sigma}_{t}-\Sigma_{t}\|_{2}$ from Lemma 5. This is why we have $1/p^{4}$ in the regret bound. We believe that there may be some room to improve the dependency of $1/p^{4}$ by controlling $\|\widehat{\Sigma}_{t}^{-1}\|_{2}$, which can be a future work.

A.5 Algorithm 3 and its regret bound

We first define some notations in Algorithm 3. Let $\lambda=Cd\log(KT)$, $V_{t}=\lambda I_{d+1}+\sum_{s=\tau^{\prime}+1}^{t}\widehat{z}_{a_{s},s}\widehat{z}_{a_{s},s}^{\top}$, and $\widehat{\theta}_{t}=V_{t}^{-1}\sum_{s=\tau^{\prime}+1}^{t}\widehat{z}_{a_{s},s}^{\top}y_{s}$ for some sufficiently large constant $C>0$. We note that $\widehat{z}_{a_{s},s}$ only for $s>\tau^{\prime}$ are used in $V_{t}$ and $\widehat{\theta}_{t}$. We also define the confidence set for estimating a latent parameter $\overline{\theta}$ as

[TABLE]

In Algorithm 1, $\widehat{z}_{a_{s},s}$ in $V_{t}$ and $\widehat{\theta}_{t}$ is required to be updated after time step $s$ because estimated features in early stages may not be accurate. However, by introducing an explicit exploration phase for the first $\tau^{\prime}$ time steps, Algorithm 3 does not use the early estimated features for $V_{t}$ and $\widehat{\theta}_{t}$. Therefore, in Algorithm 3, $\widehat{z}_{a_{s},s}$ in $V_{t}$ and $\widehat{\theta}_{t}$ is computed at time $s$ and it will not be updated anymore in later time steps.

The algorithm introduces an explicit exploration phase over $\tau^{\prime}$ time steps, which guarantees the well-estimated features after the phase. Therefore, it does not require updating estimated features for previously selected arms, which is the main bottleneck of the Algorithm 1. However, setting a proper $\tau^{\prime}$ for the exploration phase requires knowing an $\alpha$ that satisfies $\alpha\leq p$, and the regret bound depends on $\alpha$. By setting $\tau^{\prime}=\lceil d(\log(T))^{2}/(K\alpha^{4})\rceil$, we can show that the regret bound of the efficient algorithm is $R(T)=\tilde{O}\left(d\sqrt{T}+(d^{2}/p^{3/2})\sqrt{T/K}+d/(\alpha^{4}K)\right).$ The last term in the regret bound is larger than that in the regret of Theorem 2 due to the additional exploration phase in the efficient algorithm.

Now we provide a proof for the regret bound.

Regret bound of Algorithm 3

From Lemma 3, regret from $t=1$ to $\tau^{\prime}$ is bounded as

[TABLE]

Then from Lemma 5, for any time $t>\tau^{\prime}$, with probability at least $1-1/T$, we have $\|\Sigma-\widehat{\Sigma}_{t}\|_{2}=o(1)$, which incurs to have

[TABLE]

Therefore, for all $t>\tau^{\prime}$, we have

[TABLE]

Since we can get the bound for the estimated features after $\tau^{\prime}$, the algorithm utilizes the estimated features of chosen arms only after $\tau^{\prime}$ for getting $V_{t}$ and $\widehat{\theta}_{t}$. Then by following the proof steps in Theorem 2, we can easily get

[TABLE]

A.6 Real-world experiments

Here we present numerical results for real-world datasets. We compare our algorithm with OFUL and the random policy for click-through rate (CTR) at different time steps. For the comparison, we use Avazu CTR (Avazu, 2014), Taobao.com (Alibaba, 2018), and MovieLens 100K (Harper and Konstan, 2015) datasets that contain CTR or rating information collected from advertising or movie recommendation systems. We use autoencoder models to preprocess feature information, constructing item feature vectors for each item. We set preprocessed feature dimension $d=32$, number of available items at each time, $K=20$, and time horizon $T=10^{5}$. We erase some entries in the preprocessed feature vectors using masking vectors with missing probability $1-p=0.1$. We examine average CTR for each algorithm over time steps $t\in[T]$. In Figure 3, we observe that our algorithm outperforms OFUL and the random policy for all the datasets without generating both feature and noise vectors from Gaussian distributions externally. In what follows, we describe the detailed experiment settings for real-world datasets.

Avazu CTR. This dataset contains mobile advertisement recommendation log data. Each data contains user, advertisement (ad), and click information. Each user-ad pair feature contains information about device type, site category, category of visited website, banner position, etc. For modeling reward payoffs, we use click information for each recommendation (user-ad); 0 for non-click and 1 for click. Using a pre-trained autoencoder model, we preprocess each user-ad feature vector to reduce dimension to $32$ (output of the encoder), which we refer to as an item feature. The autoencoder is trained using user-ad feature information without reward information. Then, we erase each entry of the preprocessed item features with probability $0.1$ for modeling the missing data scenario. We then divide the items (user-ad pairs) into two sets according to whether the ads are clicked or not; randomly select 1000 items to construct each set. For each time, algorithms get an available item set, which is constructed by randomly selecting $K=20$ items from the two sets. In detail, the available item set consists of one selected from the clicked set and the others selected from the non-clicked set. Then, there must be one best item in the available item set. At each time, an algorithm selects an item from the available item set and get a reward according to the click information [math] or $1$. Selecting an item can be represented as recommending an ad to a user.

Taobao.com. This dataset contains advertisement display/click log data on the website of Taobao.com. As in the Avazu dataset, each item (user-ad pair) contains user and ad information such as gender, age, consumption grade, brand, category, etc. Also there exist click data $1$ or [math] for each recommendation. For each item feature, using a pre-trained autoencoder, we get a prepocessed feature vector with dimension $32$. The rest of the experiment setting is the same as the case for the Avazu CTR dataset.

MovieLens 100K. This dataset contains movie ratings from users collected through the MovieLens website. Each rating has an integer value from $1$ to $5$. For modeling binary reward payoffs, we treat rating $5$ as reward $1$ and otherwise [math]. Each user and movie data contain feature information such as age, gender, movie genre, etc. The rest of the experiment setting is the same as the experiment settings for the above datasets.