Efficient Explorative Key-term Selection Strategies for Conversational Contextual Bandits

TL;DR

This paper introduces a new framework and algorithms for conversational contextual bandits that effectively combine feedback types and incorporate explorative key-term strategies, leading to faster learning and improved accuracy.

Contribution

It proposes ConLinUCB, a unified framework for better feedback integration, and develops two algorithms with explorative key-term strategies, achieving tighter regret bounds and superior performance.

Findings

ConLinUCB-BS achieves a regret bound of O(d√T log T).

Algorithms show up to 54% improvement in learning accuracy.

Algorithms demonstrate up to 72% improvement in computational efficiency.

Abstract

Conversational contextual bandits elicit user preferences by occasionally querying for explicit feedback on key-terms to accelerate learning. However, there are aspects of existing approaches which limit their performance. First, information gained from key-term-level conversations and arm-level recommendations is not appropriately incorporated to speed up learning. Second, it is important to ask explorative key-terms to quickly elicit the user's potential interests in various domains to accelerate the convergence of user preference estimation, which has never been considered in existing works. To tackle these issues, we first propose ``ConLinUCB", a general framework for conversational bandits with better information incorporation, combining arm-level and key-term-level feedback to estimate user preference in one step at each time. Based on this framework, we further design two bandit algorithms with explorative key-term selection strategies, ConLinUCB-BS and ConLinUCB-MCR. We prove tighter regret upper bounds of our proposed algorithms. Particularly, ConLinUCB-BS achieves a regret bound of $O(d\sqrt{T\log T})$, better than the previous result $O(d\sqrt{T}\log T)$. Extensive experiments on synthetic and real-world data show significant advantages of our algorithms in learning accuracy (up to 54\% improvement) and computational efficiency (up to 72\% improvement), compared to the classic ConUCB algorithm, showing the potential benefit to recommender systems.

Tables

Table 1. Table 1: Total runninng time (in seconds) of algorithms on Movielens with T = 5 , 000 𝑇 5 000 T=5,000 .

Alogrithm	Total time	Total time for selecting arms	Total time for selecting key-terms
ConUCB	11,297	5,217	6,080
ConLinUCB-UCB	5,738	3,060	2,678
ConLinUCB-MCR	4,821	3,030	1,791
ConLinUCB-BS	3,127	3,120	6

Equations

$$r_{a_t,t}={\mathbf{x}}_{a_t}^{\top }{\mathbf{\theta }}^{\ast }+{ϵ}_t,$$

$$R(T)=\sum _{t=1}^T{\mathbf{x}}_{a_t^{\ast }}^{\top }{\mathbf{\theta }}^{\ast }-\sum _{t=1}^T{\mathbf{x}}_{a_t}^{\top }{\mathbf{\theta }}^{\ast }.$$

$${\tilde{r}}_{k,t}={\tilde{\mathbf{x}}}_k^{\top }{\mathbf{\theta }}^{\ast }+{\widetilde{ϵ}}_t,$$

$${\mathbf{\theta }}_t$$

$$+\beta {\Vert \mathbf{\theta }\Vert }_2^2,$$

$${\mathbf{\theta }}_t={\mathbf{M}}_t^{-1}{\mathbf{b}}_t,$$

$$\begin{array}{rl}{\mathbf{M}}_t& =\sum _{\tau =1}^{t-1}{\mathbf{x}}_{a_{\tau }}{\mathbf{x}}_{a_{\tau }}^{\top }+\sum _{\tau =1}^t\sum _{k\in {\mathcal{K}}_{\tau }}{\tilde{\mathbf{x}}}_k{\tilde{\mathbf{x}}}_k^{\top }+\beta \mathbf{I},\\ {\mathbf{b}}_t& =\sum _{\tau =1}^{t-1}{\mathbf{x}}_{a_{\tau }}{r}_{a_{\tau },\tau }+\sum _{\tau =1}^t\sum _{k\in {\mathcal{K}}_{\tau }}{\tilde{\mathbf{x}}}_k{\tilde{r}}_{k,\tau }.\end{array}$$

$$a_t=\underset{a\in {\mathcal{A}}_t}{\mathrm{argmax}}\underset{{\hat{R}}_{a,t}}{\underbrace{{\mathbf{x}}_a^{\top }{\mathbf{\theta }}_t}}+\underset{C_{a,t}}{\underbrace{{\alpha }_t{\Vert {\mathbf{x}}_a\Vert }_{{\mathbf{M}}_t^{-1}}}},$$

$${\alpha }_t=\sqrt{2\log (\frac{1}{\delta })+d\log (1+\frac{t+b(t)}{\beta d})}+\sqrt{\beta },$$

$$k\sim \text{unif}(\mathcal{B}),$$

$$k\in \underset{k\in {\mathcal{K}}_t}{\mathrm{argmax}}{\alpha }_t{\Vert {\tilde{\mathbf{x}}}_k\Vert }_{{\mathbf{M}}_t^{-1}},$$

$$\left|{\mathbf{x}}_a^{\top }({\mathbf{\theta }}_t-{\mathbf{\theta }}^{\ast })\right|\le {\alpha }_t{\Vert {\mathbf{x}}_a\Vert }_{{\mathbf{M}}_t^{-1}}=C_{a,t},$$

$${\lambda }_{\mathcal{B}}:={\lambda }_{\min }({\mathbf{E}}_{k\sim \text{unif}(\mathcal{B})}[{\tilde{\mathbf{x}}}_k{\tilde{\mathbf{x}}}_k^{\top }])>0,$$

$${\Vert {\mathbf{x}}_a\Vert }_{{\mathbf{M}}_t^{-1}}\le \sqrt{\frac{2}{{\lambda }_{\mathcal{B}}bt}}.$$

$$R(T)$$

$$\displaystyle\quad+\sqrt{\beta}\Bigg{)}+\frac{256}{b\lambda_{\mathcal{B}}^{2}}\log(\frac{256d}{\lambda_{\mathcal{B}}^{2}\delta})+1\,.$$

$$R(T)$$

$$+\sqrt{\frac{1-\lambda }{\lambda \beta }}\sqrt{2\log (\frac{2}{\delta })+d\log (1+\frac{bT}{\beta d})}$$

$$\displaystyle\quad\ +\sqrt{2\log(\frac{2}{\delta})+d\log(1+\frac{\lambda T}{(1-\lambda)d})}\Bigg{)}\,,$$

$$R(T)$$

$$\displaystyle\times\Bigg{(}\sqrt{\beta}+\sqrt{2\log(\frac{1}{\delta})+d\log(1+\frac{(b+1)T}{\beta d})}\Bigg{)}\,.$$

$${\mathbf{\theta }}_t-{\mathbf{\theta }}_{\ast }$$

$$\displaystyle=\Bigg{(}\sum_{\tau=1}^{t-1}\bm{x}_{a_{\tau}}\bm{x}^{\top}_{a_{\tau}}+\sum_{\tau=1}^{t}\sum_{k\in\mathcal{K}_{\tau}}\bm{\tilde{x}}_{k}\bm{\tilde{x}}^{\top}_{k}+\beta\bm{I}\Bigg{)}^{-1}\Bigg{(}\sum_{\tau=1}^{t-1}\bm{x}_{a_{\tau}}r_{a_{\tau},\tau}+\sum_{\tau=1}^{t}\sum_{k\in\mathcal{K}_{\tau}}\bm{\tilde{x}}_{k}\tilde{r}_{k,\tau}\Bigg{)}-\bm{\theta}_{*}$$

$$\displaystyle=\Bigg{(}\sum_{\tau=1}^{t-1}\bm{x}_{a_{\tau}}\bm{x}^{\top}_{a_{\tau}}+\sum_{\tau=1}^{t}\sum_{k\in\mathcal{K}_{\tau}}\bm{\tilde{x}}_{k}\bm{\tilde{x}}^{\top}_{k}+\beta\bm{I}\Bigg{)}^{-1}\Bigg{(}\sum_{\tau=1}^{t-1}\bm{x}_{a_{\tau}}\bigg{(}\bm{x}^{\top}_{a_{\tau}}\bm{\theta}_{*}+\epsilon_{\tau}\bigg{)}+\sum_{\tau=1}^{t}\sum_{k\in\mathcal{K}_{\tau}}\bm{\tilde{x}}_{k}\bigg{(}\bm{\tilde{x}}_{k}^{\top}\bm{\theta}_{*}+\tilde{\epsilon}_{\tau}\bigg{)}\Bigg{)}-\bm{\theta}_{*}$$

$$\displaystyle=\Bigg{(}\sum_{\tau=1}^{t-1}\bm{x}_{a_{\tau}}\bm{x}^{\top}_{a_{\tau}}+\sum_{\tau=1}^{t}\sum_{k\in\mathcal{K}_{\tau}}\bm{\tilde{x}}_{k}\bm{\tilde{x}}^{\top}_{k}+\beta\bm{I}\Bigg{)}^{-1}\Bigg{(}\sum_{\tau=1}^{t-1}\bm{x}_{a_{\tau}}\bm{x}^{\top}_{a_{\tau}}+\sum_{\tau=1}^{t}\sum_{k\in\mathcal{K}_{\tau}}\bm{\tilde{x}}_{k}\bm{\tilde{x}}^{\top}_{k}+\beta\bm{I}-\beta\bm{I}\Bigg{)}\bm{\theta}_{*}-\bm{\theta}_{*}$$

$$+{\mathbf{M}}_t^{-1}(\sum _{\tau =1}^{t-1}{\mathbf{x}}_{a_{\tau }}{ϵ}_{\tau }+\sum _{\tau =1}^t\sum _{k\in {\mathcal{K}}_{\tau }}{\tilde{\mathbf{x}}}_k{\widetilde{ϵ}}_{\tau })$$

$$=-\beta {\mathbf{M}}_t^{-1}{\mathbf{\theta }}_{\ast }+{\mathbf{M}}_t^{-1}(\sum _{\tau =1}^{t-1}{\mathbf{x}}_{a_{\tau }}{ϵ}_{\tau }+\sum _{\tau =1}^t\sum _{k\in {\mathcal{K}}_{\tau }}{\tilde{\mathbf{x}}}_k{\widetilde{ϵ}}_{\tau }).$$

$$\left|{\mathbf{x}}_a^{\top }({\mathbf{\theta }}_t-{\mathbf{\theta }}_{\ast })\right|$$

$$\le \beta {\Vert {\mathbf{x}}_a^{\top }{\mathbf{M}}_t^{-\frac{1}{2}}\Vert }_2{\Vert {\mathbf{M}}_t^{-\frac{1}{2}}{\mathbf{\theta }}_{\ast }\Vert }_2+{\Vert {\mathbf{x}}_a^{\top }{\mathbf{M}}_t^{-\frac{1}{2}}\Vert }_2\times {\Vert {\mathbf{M}}_t^{-\frac{1}{2}}(\sum _{\tau =1}^{t-1}{\mathbf{x}}_{a_{\tau }}{ϵ}_{\tau }+\sum _{\tau =1}^t\sum _{k\in {\mathcal{K}}_{\tau }}{\tilde{\mathbf{x}}}_k{\widetilde{ϵ}}_{\tau })\Vert }_2$$

$$\le \beta {\Vert {\mathbf{x}}_a\Vert }_{{\mathbf{M}}_t^{-1}}{\Vert {\mathbf{M}}_t^{-\frac{1}{2}}\Vert }_2{\Vert {\mathbf{\theta }}_{\ast }\Vert }_2+{\Vert {\mathbf{x}}_a\Vert }_{{\mathbf{M}}_t^{-1}}{\Vert \sum _{\tau =1}^{t-1}{\mathbf{x}}_{a_{\tau },\tau }{ϵ}_{\tau }+\sum _{\tau =1}^t\sum _{k\in {\mathcal{K}}_{\tau }}{\tilde{\mathbf{x}}}_k{\widetilde{ϵ}}_{\tau }\Vert }_{{\mathbf{M}}_t^{-1}}$$

$$\displaystyle\leq\left\|\bm{x}_{a}\right\|_{\bm{M}_{t}^{-1}}\Bigg{(}\sqrt{\beta}\left\|\bm{\theta}_{*}\right\|_{2}+\left\|\sum_{\tau=1}^{t-1}\bm{x}_{a_{\tau}}\epsilon_{\tau}+\sum_{\tau=1}^{t}\sum_{k\in\mathcal{K}_{\tau}}\bm{\tilde{x}}_{k}\tilde{\epsilon}_{\tau}\right\|_{\bm{M}_{t}^{-1}}\Bigg{)}\,,$$

$$\left\|\sum_{\tau=1}^{t-1}\bm{x}_{a_{\tau}}\epsilon_{\tau}+\sum_{\tau=1}^{t}\sum_{k\in\mathcal{K}_{\tau}}\bm{\tilde{x}}_{k}\tilde{\epsilon}_{k}\right\|_{\bm{M}_{t}^{-1}}\leq\sqrt{2\log\bigg{(}\frac{det(\bm{M}_{t})^{\frac{1}{2}}det(\beta\bm{I})^{\frac{1}{2}}}{\delta}\bigg{)}}\,,$$

$$det({\mathbf{M}}_t)=\prod _{i=1}^d{\lambda }_i$$

$$\displaystyle=\big{(}\frac{trace(\bm{M}_{t})}{d}\big{)}^{d}$$

$$\displaystyle=\Bigg{(}\frac{trace(\sum_{\tau=1}^{t-1}\bm{x}_{a_{\tau}}\bm{x}^{\top}_{a_{\tau}}+\sum_{\tau=1}^{t}\sum_{k\in\mathcal{K}_{\tau}}\bm{\tilde{x}}_{k}\bm{\tilde{x}}^{\top}_{k}+\beta\bm{I})}{d}\Bigg{)}^{d}\leq\big{(}\frac{t+b(t)+\beta d}{d}\big{)}^{d}\,,$$

$${\Vert \sum _{\tau =1}^{t-1}{\mathbf{x}}_{a_{\tau }}{ϵ}_{\tau }+\sum _{\tau =1}^t\sum _{k\in {\mathcal{K}}_{\tau }}{\tilde{\mathbf{x}}}_k{\widetilde{ϵ}}_k\Vert }_{{\mathbf{M}}_t^{-1}}\le \sqrt{2\log (\frac{1}{\delta })+d\log (1+\frac{b(t)+t}{\beta d})}.$$

$${\lambda }_{\mathcal{B}}:={\lambda }_{\min }({\mathbf{E}}_{k\sim \text{unif}(\mathcal{B})}[{\tilde{\mathbf{x}}}_k{\tilde{\mathbf{x}}}_k^{\top }])>0.$$

$${\lambda }_{\min }(\sum _{\tau =1}^t\sum _{k\in {\mathcal{K}}_{\tau }}{\tilde{\mathbf{x}}}_k{\tilde{\mathbf{x}}}_k^{\top })\ge \frac{{\lambda }_{\mathcal{B}}bt}{2},$$

Full text

1 Introduction

Nowadays, recommender systems are widely used in various areas. The learning speed for traditional online recommender systems is usually slow since extensive exploration is needed to discover user preferences. To accelerate the learning process and provide more personalized recommendations, the conversational recommender system (CRS) has been proposed (Christakopoulou, Radlinski, and Hofmann 2016; Christakopoulou et al. 2018; Sun and Zhang 2018; Zhang et al. 2018; Li et al. 2021; Gao et al. 2021). In a CRS, a learning agent occasionally asks for the user’s explicit feedback on some “key-terms”, and leverages this additional conversational information to better elicit the user’s preferences (Zhang et al. 2020; Xie et al. 2021).

Despite the recent success of CRS, there are crucial limitations in using conversational contextual bandit approaches to design recommender systems. These limitations include: (a) The information gained from key-term-level conversations and arm-level recommendations is not incorporated properly to speed up learning, as the user preferences are essentially assumed to be the same in these two stages but are estimated separately (Zhang et al. 2020; Xie et al. 2021; Wu et al. 2021); (b) Queries using traditional key-terms were restrictive and not explorative enough. Specifically, we say a key-term is “explorative” if it is under-explored so far and the system is uncertain about the user’s preferences in its associated items. Asking for the user’s feedback on explorative key-terms can efficiently elicit her potential interests in various domains (e.g., sports, science), which means we can quickly estimate the user preference vector in all directions of the feature space, thus accelerating the learning speed. Therefore, it is crucial to design explorative key-term selection strategies, which existing works have not considered.

Motivated by the above considerations, we propose to design conversational bandit algorithms that (i) estimate the user’s preferences utilizing both arm-level and key-term-level interactions simultaneously to properly incorporate the information gained from both two levels and (ii) use effective strategies to choose explorative key-terms when conducting conversations for quick user preference inference.

To better utilize the interactive feedback from both recommendations and conversations, we propose ConLinUCB, a general framework for conversational bandits with possible flexible key-term selection strategies. ConLinUCB estimates the user preference vector by solving one single optimization problem that minimizes the mean squared error of both arm-level estimated rewards and key-term-level estimated feedback simultaneously, instead of separately estimating at different levels as in previous works. In this manner, the information gathered from these two levels can be better combined to guide the learning.

Based on this ConLinUCB framework, we design two new algorithms with explorative key-term selection strategies, ConLinUCB-BS and ConLinUCB-MCR.

• •

ConLinUCB-BS makes use of a barycentric spanner containing linearly independent vectors, which can be an efficient exploration basis in bandit problems (Amballa, Gupta, and Bhat 2021). Whenever a conversation is allowed, ConLinUCB-BS selects an explorative key-term uniformly at random from a precomputed barycentric spanner $\mathcal{B}$ of the given key-term set $\mathcal{K}$.

• •

ConLinUCB-MCR applies in a more general setting when the key-term set can be time-varying, and it can leverage interactive histories to choose explorative key-terms adaptively. Note that in the bandit setting, we often use confidence radius to adaptively evaluate whether an arm has been sufficiently explored, and the confidence radius of an arm will shrink whenever it is selected (Lattimore and Szepesvári 2020). This implies that an explorative key-term should have a large confidence radius. Based on this reasoning, ConLinUCB-MCR selects the most explorative key-terms with maximal confidence radius when conducting conversations.

Equipped with explorative conversations, our algorithms can quickly elicit user preferences for better recommendations. For example, if the key-term sports is explorative at round $t$, indicating that so far the agent is not sure whether the user favors items associated with sports (e.g., basketball, volleyball), it will ask for the user’s feedback on sports directly and conduct recommendations accordingly. In this manner, the agent can quickly find suitable items for the user. We prove the regret upper bounds of our algorithms, which are better than the classic ConUCB algorithm.

In summary, our paper makes the following contributions:

• •

We propose a new and general framework for conversational contextual bandits, ConLinUCB, which can efficiently incorporate the interactive information gained from both recommendations and conversations.

• •

Based on ConLinUCB, we design two new algorithms with explorative key-term selection strategies, ConLinUCB-BS and ConUCB-MCR, which can accelerate the convergence of user preference estimation.

• •

We prove that our algorithms achieve tight regret upper bounds. Particularly, ConLinUCB-BS achieves a bound of $O(d\sqrt{T\log T})$, better than the previous $O(d\sqrt{T}\log T)$ in the conversational bandits literature.

• •

Experiments on both synthetic and real-world data validate the advantages of our algorithms in both learning accuracy (up to 54% improvement) and computational efficiency (up to 72% improvement)111Codes are available at https://github.com/ZhiyongWangWzy/ConLinUCB..

2 Related Work

Our work is most closely related to the research on conversational contextual bandits.

Contextual linear bandit is an online sequential decision-making problem where at each time step, the agent has to choose an action and receives a corresponding reward whose expected value is an unknown linear function of the action (Li et al. 2010; Chu et al. 2011; Abbasi-Yadkori, Pál, and Szepesvári 2011; Wu et al. 2016). The objective is to collect as much reward as possible in $T$ rounds.

Traditional linear bandits need extensive exploration to capture the user preferences in recommender systems. To speed up online recommendations, the idea of conversational contextual bandits was first proposed in (Zhang et al. 2020), where conversational feedback on key-terms is leveraged to assist the user preference elicitation. In that work, they propose the ConUCB algorithm with a theoretical regret bound of $O(d\sqrt{T}\log T)$. Some follow-up works try to improve the performance of ConUCB with the help of additional information, such as self-generated key-terms (Wu et al. 2021), relative feedback (Xie et al. 2021), and knowledge graph (Zhao et al. 2022). Unlike these works, we adopt the same problem settings as ConUCB and improve the underlying mechanisms without relying on additional information. Yet one can use the principles of efficient information incorporation and explorative conversations proposed in this work to enhance these works when additional information is available, which is left as an interesting future work.

3 Problem Settings

This section states the problem setting of conversational contextual bandits. Suppose there is a finite set $\mathcal{A}$ of arms. Each arm $a\in\mathcal{A}$ represents an item to be recommended and is associated with a feature vector $\bm{x}_{a}\in\mathbb{R}^{d}$. Without loss of generality, the feature vectors are assumed to be normalized, i.e., $\left\|\bm{x}_{a}\right\|_{2}=1$, ${\forall}a\in\mathcal{A}$. The agent interacts with a user in $T\in\mathbb{N}_{+}$ rounds, whose preference of items is represented by an unknown vector $\bm{\theta}^{*}\in\mathbb{R}^{d}$, $\left\|\bm{\theta}^{*}\right\|_{2}\leq 1$.

At each round $t=1,2,...,T$, a subset of arms $\mathcal{A}_{t}\subseteq\mathcal{A}$ are available to the agent to choose from. Based on historical interactions, the agent selects an arm $a_{t}\in\mathcal{A}_{t}$, and receives a corresponding reward $r_{a_{t},t}\in[0,1]$. The reward is assumed to be a linear function of the contextual vectors

[TABLE]

where $\epsilon_{t}$ is 1-sub-Gaussian random noise with zero mean.

Let $a_{t}^{*}\in{\arg\max}_{a\in\mathcal{A}_{t}}\bm{x}_{a}^{\top}\bm{\theta}^{*}$ denote an optimal arm with the largest expected reward at $t$. The learning objective is to minimize the cumulative regret

[TABLE]

The agent can also occasionally query the user’s feedback on some conversational key-terms to help elicit user preferences. In particular, a “key-term” is a keyword or topic related to a subset of arms. For example, the key-term sports is related to the arms like basketball, football, swimming, etc.

Suppose there is a finite set $\mathcal{K}$ of key-terms. The relationship between arms and key-terms is given by a weighted bipartite graph $(\mathcal{A},\mathcal{K},\bm{W})$, where $\bm{W}\triangleq\left[w_{a,k}\right]_{a\in\mathcal{A},k\in\mathcal{K}}$ represents the relationship between arms and key-terms, i.e., a key-term $k\in\mathcal{K}$ is associated to an arm $a\in\mathcal{A}$ with weight $w_{a,k}\geq 0$. We assume that each key-term $k$ has positive weights with some related arms (i.e., $\sum_{a\in\mathcal{A}}w_{a,k}>0$, $\forall{k\in\mathcal{K}}$), and the weights associated with each arm sum up to 1, i.e., $\sum_{k\in\mathcal{K}}w_{a,k}=1$, $a\in\mathcal{A}$. The feature vector of a key-term $k$ is given by $\tilde{\bm{x}}_{k}=\sum_{a\in\mathcal{A}}\frac{w_{a,k}}{\sum_{a^{\prime}\in\mathcal{A}}w_{a^{\prime},k}}\bm{x}_{a}$. The key-term-level feedback on the key-term $k$ at $t$ is defined as

[TABLE]

where $\tilde{\epsilon}_{t}$ is assumed to be 1-sub-Gaussian random noise. One thing to stress is that in the previous works (Zhang et al. 2020; Wu et al. 2021; Xie et al. 2021; Zhao et al. 2022), the unknown user preference vector $\bm{\theta}^{*}$ is essentially assumed to be the same at both the arm level and the key-term level.

To avoid affecting the user experience, the agent should not conduct conversations too frequently. Following (Zhang et al. 2020), we define a function $b:\mathbb{N}_{+}\rightarrow\mathbb{R}_{+}$, where $b(t)$ is increasing in $t$, to control the conversation frequency of the agent. At each round $t$, if $b(t)-b(t-1)>0$, the agent is allowed to conduct $q(t)=\lfloor b(t)-b(t-1)\rfloor$ conversations by asking for user’s feedback on $q(t)$ key-terms. Using this modeling arrangement, the agent will have $b(t)$ conversational interactions with the user up to round $t$.

4 Algorithms and Theoretical Analysis

This section first introduces ConLinUCB, a framework for conversational bandits with better information incorporation, which is general for “any” key-term selection strategies. Based on ConLinUCB, we further propose two bandit algorithms, ConLinUCB-BS and ConLinUCB-MCR, with explorative key-term selection strategies.

To simplify the exposition, we merge the ConLinUCB framework, ConLinUCB-BS and ConLinUCB-MCR in Algorithm 1. We also theoretically give regret bounds of our proposed algorithms.

4.1 General ConLinUCB Algorithm Framework

In conversational bandits, it is common that the unknown preference vector $\bm{\theta}^{*}$ is essentially assumed to be the same at both arm level and key-term level (Zhang et al. 2020; Xie et al. 2021; Wu et al. 2021). However, all existing works treat $\bm{\theta}^{*}$ differently at these two levels. Specifically, they take two different steps to estimate user preference vectors at the arm level and key-term level, and use a discounting parameter $\lambda\in(0,1)$ to balance learning from these two levels’ interactions. In this manner, the contributions of the arm-level and key-term-level information to the convergence of estimation are discounted by $\lambda$ and $1-\lambda$, respectively. Therefore, such discounting will cause waste of observations, indicating that information at these two levels can not be fully leveraged to accelerate the learning process.

To handle the above issues, we propose a general framework called ConLinUCB, for conversational contextual bandits. In this new framework, in order to fully leverage interactive information from two levels, we simultaneously estimate the user preference vector by solving one single optimization problem that minimizes the mean squared error of both arm-level estimated rewards and key-term-level estimated feedback. Specifically, in ConLinUCB, at round $t$, the user preference vector is estimated by solving the following linear regression

[TABLE]

where $\mathcal{K}_{\tau}$ denotes the set of key-terms asked at round $\tau$, and the coefficient $\beta>0$ controls regularization. The closed-form solution of this optimization problem is

[TABLE]

where

[TABLE]

To balance exploration and exploitation, ConLinUCB selects arms using the upper confidence bound (UCB) strategy

[TABLE]

where $\left\|\bm{x}\right\|_{\bm{M}}=\sqrt{\bm{x}^{\top}\bm{M}\bm{x}}$, $\hat{R}_{a,t}$ and $C_{a,t}$ denote the estimated reward and confidence radius of arm $a$ at round $t$, and

[TABLE]

which comes from the following Lemma 1.

The ConLinUCB algorithm framework is shown in Alg. 1. The key-term-level interactions take place in line 3-14. At round $t$, the agent first determines whether conversations are allowed using $b(t)$. When conducting conversations, the agent asks for the user’s feedback on $q(t)$ key-terms and uses the feedback to update the parameters. Line 15-20 summarise the arm-level interactions. Based on historical interactions, the agent calculates the estimated $\bm{\theta}^{*}$, selects an arm with the largest UCB index, receives the corresponding reward, and updates the parameters accordingly. ConLinUCB only maintains one set of covariance matrix $\bm{M}_{t}$ and regressand vector $\bm{b}_{t}$, containing the feedback from both arm-level and key-term-level interactions. By doing so, ConLinUCB better leverages the feedback information than ConUCB. Note that ConLinUCB is a general framework with the specified key-term selection strategy $\bm{\pi}$ to be determined.

4.2 ConLinUCB with key-terms from Barycentric Spanner (ConLinUCB-BS)

Based on the ConLinUCB framework, we propose the ConLinUCB-BS algorithm with an explorative key-term selection strategy. Specifically, ConLinUCB-BS selects key-terms from the barycentric spanner $\mathcal{B}$ of the key-term set $\mathcal{K}$, which is an efficient exploration basis in online learning (Amballa, Gupta, and Bhat 2021), to conduct explorative conversations. Below is the formal definition of the barycentric spanner for the key-term set $\mathcal{K}$.

Definition 1** (Barycentric Spanner of $\mathcal{K}$)**

A subset $\mathcal{B}=\{k_{1},k_{2},...,k_{d}\}\subseteq\mathcal{K}$ is a barycentric spanner for $\mathcal{K}$ if for any $k\in\mathcal{K}$, there exists a set of coefficients $\bm{c}\in[-1,1]^{d}$, such that $\tilde{\bm{x}}_{k}=\sum_{i=1}^{d}\bm{c}_{i}\tilde{\bm{x}}_{k_{i}}$.

We assume that the key-term set $\mathcal{K}$ is finite and $\{\tilde{\bm{x}}_{k}\}_{k\in\mathcal{K}}$ span $\mathbb{R}^{d}$, thus the existence of a barycentric spanner $\mathcal{B}$ of $\mathcal{K}$ is guaranteed (Awerbuch and Kleinberg 2008).

Corresponding vectors in the barycentric spanner are linearly independent. By choosing key-terms from the barycentric spanner, we can quickly explore the unknown user preference vector $\bm{\theta}^{*}$ in various directions. Based on this reasoning, whenever a conversation is allowed, ConLinUCB-BS selects a key-term

[TABLE]

which means sampling a key-term $k$ uniformly at random from the barycentric spanner $\mathcal{B}$ of $\mathcal{K}$. ConLinUCB-BS is completed using the above strategy as $\bm{\pi}$ in the ConLinUCB framework (Alg. 1). As shown in the following Lemma 1 and Lemma 2, in ConLinUCB-BS, the statistical estimation uncertainty shrinks quickly. Additionally, since the barycentric spanner $\mathcal{B}$ of the key-term set $\mathcal{K}$ can be precomputed offline, ConLinUCB-BS is computationally efficient, which is vital for real-time recommendations.

4.3 ConLinUCB with key-terms having Max Confidence Radius (ConLinUCB-MCR)

We can further improve ConLinUCB-BS in the following aspects. First, ConLinUCB-BS does not apply in a more general setting where the key-term set $\mathcal{K}$ varies over time since it needs a precomputed barycentric spanner $\mathcal{B}$ of $\mathcal{K}$. Second, as the selection of key-terms is independent of past observations, ConLinUCB-BS does not fully leverage the historical information. For example, suppose the agent is already certain about whether the user favors sports based on previous interactions. In that case, it does not need to ask for the user’s feedback on the key-term sports anymore. To address these issues, we propose the ConLinUCB-MCR algorithm that (i) is applicable when the key-term set $\mathcal{K}$ varies with $t$ and (ii) can adaptively conduct explorative conversations based on historical interactions.

In multi-armed bandits, confidence radius is used to capture whether an arm has been well explored in the interactive history, and it will shrink whenever the arm is selected. Motivated by this, if a key-term has a large confidence radius, it means the system has not sufficiently explored the user’s preferences in its related items, indicating that this key-term is explorative. Based on this reasoning, ConLinUCB-MCR selects key-terms with maximal confidence radius to conduct explorative conversations apdaptively. Specifically, when a conversation is allowed at $t$, ConLinUCB-MCR chooses a key-term as follow

[TABLE]

where $\alpha_{t}$ is defined in Eq. (8) and $\mathcal{K}_{t}\subseteq\mathcal{K}$ denotes the possibly time-varying key-terms set available at round $t$. ConLinUCB-MCR is completed using the above strategy (Eq. (10)) as $\bm{\pi}$ in ConLinUCB (Alg. 1).

4.4 Theoretical Analysis

We give upper bounds of the regret for our algorithms. As a convention, the conversation frequency satisfies $b(t)\leq t$, so we assume $b(t)=b\cdot t$, $b\in(0,1)$. We leave the proofs of Lemma 1-2 and Theorem 3-4 to the Appendix due to the space limitation.

The following lemma shows a high probability upper bound of the difference between $\bm{\theta}_{t}$ and $\bm{\theta}^{*}$ in the direction of the action vector $\bm{x}_{a}$ for algorithms based on ConLinUCB.

Lemma 1

At ${\forall}t$, for any $a\in\mathcal{A}$, with probability at least $1-\delta$ for some $\delta\in(0,1)$

[TABLE]

where $\alpha_{t}=\sqrt{2\log(\frac{1}{\delta})+d\log(1+\frac{t+b(t)}{\beta d})}+\sqrt{\beta}$.

For a barycentric spanner $\mathcal{B}$ of the key-term set $\mathcal{K}$, let

[TABLE]

where $\lambda_{\min}(\cdot)$ denotes the minimum eigenvalue of the augment. We can get the following Lemma that gives a high probability upper bound of $\left\|\bm{x}_{a}\right\|_{\bm{M}_{t}^{-1}}$ for ConLinUCB-BS.

Lemma 2

For ConLinUCB-BS, ${\forall}a\in\mathcal{A}$, at ${\forall}t\geq t_{0}=\frac{256}{b\lambda_{\mathcal{B}}^{2}}\log(\frac{128d}{\lambda_{\mathcal{B}}^{2}\delta})$, with probability at least $1-\delta$ for $\delta\in(0,\frac{1}{8}]$

[TABLE]

The following theorem gives a high probability regret upper bound of our ConLinUCB-BS.

Theorem 3

With probability at least $1-\delta$ for some $\delta\in(0,\frac{1}{4}]$, the regret $R(T)$ of ConLinUCB-BS satisfies

[TABLE]

Recall that the regret upper bound of ConUCB (Zhang et al. 2020) is

[TABLE]

which is of $O(d\sqrt{T}\log T)$. The regret bound of ConLinUCB-BS given in Theorem 3 is of $O(d\sqrt{T\log T})$ (as $\lambda_{\mathcal{B}}$ is of order $O(\frac{1}{d})$), better than ConUCB by reducing a multiplicative $\sqrt{\log T}$ term.

Next, the following theorem gives a high-probability regret upper bound of ConLinUCB-MCR.

Theorem 4

With probability at least $1-\delta$ for some $\delta\in(0,1)$, the regret $R(T)$ of ConLinUCB-MCR satisfies

[TABLE]

Note that the regret upper bound of ConLinUCB-MCR is smaller than ConUCB by reducing some additive terms.

5 Experiments on Synthetic Dataset

In this section, we show the experimental results on synthetic data. To obtain the offline-precomputed barycentric spanner $\mathcal{B}$, we use the method proposed in (Awerbuch and Kleinberg 2008).

5.1 Experimental Settings

Generation of the synthetic dataset.

We create a set of arms $\mathcal{A}$ with $\left|\mathcal{A}\right|=5,000$ arms, and a set of key-terms $\mathcal{K}$ with $\left|\mathcal{K}\right|=500$. We set the dimension of the feature space to be $d=50$ and the number of users $N_{u}=200$.

For each user preference vector $\bm{\theta}_{u}^{*}$ and each arm feature vector $\bm{x}_{a}$, each entry is generated by independently drawing from the standard normal distribution $\mathcal{N}(0,1)$, and all these vectors are normalized such that $\left\|\bm{\theta}_{u}^{*}\right\|_{2}=1$, $\left\|\bm{x}_{a}\right\|_{2}=1$. The weight matrix $\bm{W}\triangleq\left[w_{a,k}\right]$ is generated as follows: First, for each key-term $k$, we select an integer $n_{k}\in[1,10]$ uniformly at random, then randomly select a subset of $n_{k}$ arms $\mathcal{A}_{k}$ to be the related arms for key-term $k$; second, for each arm $a$, if it is related to a set of $n_{a}$ key-terms $\mathcal{K}_{a}$, we assign equal weights $w_{a,k}=\frac{1}{n_{a}}$, $\forall{k\in\mathcal{K}_{a}}$. Following (Zhang et al. 2020), the feature vector for each key-term $k$ is computed using $\tilde{\bm{x}}_{k}=\sum_{a\in\mathcal{A}}\frac{w_{a,k}}{\sum_{a^{\prime}\in\mathcal{A}}w_{a^{\prime},k}}\bm{x}_{a}$. The arm-level rewards and key-term-level feedback are generated following Eq. (1) and Eq. (3).

Baselines.

We compare our algorithms with the following baselines:

• •

LinUCB (Li et al. 2010): A state-of-the-art contextual linear bandit algorithm that selects arms only based on the arm-level feedback without using conversational feedback.

• •

Arm-Con (Christakopoulou, Radlinski, and Hofmann 2016): A conversational bandit algorithm that conducts conversations on arms without considering key-terms, and uses LinUCB for arm selection.

• •

ConUCB (Zhang et al. 2020): The core conversational bandit algorithm that selects a key-term to minimize some estimation error whenever a conversation is allowed.

• •

ConLinUCB-UCB: An algorithm using a LinUCB-alike method as the key-term selection strategy in our proposed ConLinUCB framework, i.e., choose key-term $k\in\mathop{\arg\max}\limits_{k\in\mathcal{K}_{t}}\bm{\tilde{x}}_{k}^{\top}\bm{\theta}_{t}+\alpha_{t}\left\|\bm{\tilde{x}}_{k}\right\|_{\bm{M}_{t}^{-1}}$ at round $t$.

5.2 Evaluation Results

This section first shows the results when the key-term set $\mathcal{K}$ is fixed. In this case, we evaluate the regret $R(T)$ for all algorithms, and we study the impact of the conversation frequency function $b(t)$ and the number of arms $\left|\mathcal{A}_{t}\right|$ available at each round $t$. When $\mathcal{K}$ varies with time, ConLinUCB-BS does not apply, and we compare the regret of other algorithms. Following (Zhang et al. 2020), we set $T=1,000$, $b(t)=5\lfloor\log(t+1)\rfloor$ and $\left|\mathcal{A}_{t}\right|=50$, unless otherwise stated.

Cumulative regret

We run the experiments 10 times and calculate the average regret of all the users for each algorithm. We include $\pm std$ as the error bar, where $std$ stands for the standard deviation. The results are given in Figure 1 (a). First, all other algorithms outperform LinUCB, showing the advantage of conversations. Further, with our proposed ConLinUCB framework, even if we use ConLinUCB-UCB with a simple LinUCB-alike key-term selection strategy, the performance is already better than ConUCB (34.91% improvement), showing more efficient information incorporation. With explorative conversations, ConLinUCB-BS and ConLinUCB-MCR achieve much lower regrets (37.00% and 43.10% improvement over ConUCB respectively), indicating better learning accuracy. ConLinUCB-MCR further leverages historical information to conduct explorative conversations adaptively, thus achieving the lowest regret.

Impact of conversation frequency function $b(t)$

A larger $b(t)$ means the agent can conduct more conversations. We set $b(t)=f_{q}\cdot\lfloor\log t\rfloor$ and vary $f_{q}$ to change the conversation frequencies, i.e., $f_{q}\in\{5,10,20,30\}$. The results are shown in Figure 1 (b). With larger $b(t)$, our algorithms have less regret, showing the power of conversations. In all cases, ConLinUCB-BS and ConLinUCB-MCR have lower regrets than ConUCB, and ConLinUCB-MCR performs the best.

Impact of $\left|\mathcal{A}_{t}\right|$

We vary $\left|\mathcal{A}_{t}\right|$ to be 25, 50, 100, 200, 500. To clearly show the advantage of our algorithms, we evaluate the difference in regrets between LinUCB and other algorithms, i.e., $R_{\text{LinUCB}}(T)-R(T)$, representing the improved accuracy of the conversational bandit algorithms as compared with LinUCB. Note that the larger $|\mathcal{A}_{t}|$ is, the harder it is for the algorithm to identify the best arm. Results in Figure 1 (c) show that as $\left|\mathcal{A}_{t}\right|$ increases, the advantages of ConLinUCB-BS and ConLinUCB-MCR become more significant. Particularly, when $|\mathcal{A}_{t}|$=25, ConLinUCB-BS and ConLinUCB-MCR achieve 34.99% and 40.21% improvement over ConUCB respectively; when $|\mathcal{A}_{t}|$=500, ConLinUCB-BS and ConLinUCB-MCR achieve 50.36% and 53.77% improvement over ConUCB, respectively. In real applications, the size of arm set $\left|\mathcal{A}_{t}\right|$ is usually very large. Therefore, our proposed algorithms are expected to significantly outperform ConUCB in practice.

Cumulative regret for time-varying $\mathcal{K}$

This section studies the case when only a subset of key-terms $\mathcal{K}_{t}\subseteq\mathcal{K}$ are available to the agent at each round $t$, where ConLinUCB-BS is not applicable as mentioned before. The number of key-terms available at each time $t$ is set to be $\left|\mathcal{K}_{t}\right|=300$. At round $t$, 300 key-terms are chosen uniformly at random from $\mathcal{K}$ to form $\mathcal{K}_{t}$. We evaluate the regret of all algorithms except ConLinUCB-BS. The results are shown in Figure 1 (d). We can observe that ConLinUCB-MCR outperforms all baselines and achieves 43.02% improvement over ConUCB.

6 Experiments on Real-world Datasets

This section shows the experimental results on two real-world datasets, Last.FM and Movielens. The baselines, generations of arm-level rewards and key-term-level feedback, and the computation method of the barycentric spanner are the same as in the last section. Following the experiments on real data of (Zhang et al. 2020), we set $T=5,000$, $b(t)=5\lfloor\log(t+1)\rfloor$ and $\left|\mathcal{A}_{t}\right|=50$, unless otherwise stated.

6.1 Experiment Settings

Last.FM and Movielens datasets (Cantador, Brusilovsky, and Kuflik 2011)

Last.FM is a dataset for music artist recommendations containing 186,479 interaction records between 1,892 users and 17,632 artists. Movielens is a dataset for movie recommendation containing 47,957 interaction records between 2,113 users and 10,197 movies.

Generation of the data

The data is generated following (Li et al. 2019; Zhang et al. 2020; Wu et al. 2021). We treat each music artist and each movie as an arm. For both datasets, we extract $\left|\mathcal{A}\right|=2,000$ arms with the most assigned tags by users and $N_{u}=500$ users who have assigned the most tags. For each arm, we keep at most 20 tags that are related to the most arms, and consider them as the associated key-terms of the arm. All the kept key-terms associated with the arms form the key-term set $\mathcal{K}$. The number of key-terms for Last.FM is $\left|\mathcal{K}\right|=2,726$ and that for Movielens is $5,585$. The weights of all key-terms related to the same arm are set to be equal. Based on the interactive recordings, the user feedback is constructed as follows: if the user has assigned tags to the item, the feedback is 1, otherwise the feedback is 0. To generate the feature vectors of users and arms, following (Li et al. 2019), we construct a feedback matrix $\bm{F}\in\mathbb{R}^{N_{u}\times N}$ based on the above user feedback, and decompose it using the singular-value decomposition (SVD): $\bm{F}=\bm{\Theta}\bm{S}\bm{X}^{\top}$, where $\bm{\Theta}=(\bm{\theta_{u}^{*}})$, $u\in[N_{u}]$ and $\bm{X}=(\bm{x}_{a})$, $a\in[N]$. We select $d=50$ dimensions with highest singular values in $\bm{S}$. Following (Zhang et al. 2020), feature vectors of key-terms are calculated using $\tilde{\bm{x}}_{k}=\sum_{a\in\mathcal{A}}\frac{w_{a,k}}{\sum_{a^{\prime}\in\mathcal{A}}w_{a^{\prime},k}}\bm{x}_{a}$. The arm-level rewards and key-term-level feedback are then generated following Eq. (1) and Eq. (3).

6.2 Evaluation Results

This section first shows the results on both datasets in two cases: $\mathcal{K}$ is fixed and $\mathcal{K}$ is varying with time $t$. We also compare the running time of all algorithms on the Movielens dataset, since it has more key-terms than Last.FM.

Cumulative regret

We run the experiments 10 times and calculate the average regret of all the users over $T=5,000$ rounds on the fixed generated datasets. The randomness of experiments comes from the randomly chosen $\mathcal{A}_{t}$ (also $\mathcal{K}_{t}$ in the varying key-term set case) and the randomness in the ConLinUCB-BS algorithm. We also include $\pm std$ as the error bar. For the time-varying key-term sets case, we set $\left|\mathcal{K}_{t}\right|=1,000$ and randomly select $\left|\mathcal{K}_{t}\right|$ key-terms from $\mathcal{K}$ to form $\mathcal{K}_{t}$ at round $t$. Results on Last.FM and Movielens for fixed key-term set are shown in Figure 2 (a) and Figure 2 (b). On both datasets, the regrets of ConLinUCB-BS and ConLinUCB-MCR are much smaller than ConUCB (13.28% and 17.12% improvement on Last.FM, 13.08% and 16.93% improvement on Movielens, respectively) and even the simple ConLinUCB-UCB based on our ConLinUCB framework outperforms ConUCB. Results on Last.FM and Movielens for varying key-term sets are given in Figure 2 (c) and Figure 2 (d). ConLinUCB-MCR performs much better than ConUCB on both datasets (19.66% and 17.85% improvement on Last.FM and Movielens respectively).

Running time

We evaluate the running time of all the conversational bandit algorithms on the representative Movielens dataset to compare their computational efficiency. For clarity, we report the total running time for selecting arms and key-terms. We set $T=5,000$ and the results are summarized in Table 1. It is clear that our algorithms cost much less time in both key-term selection and arm selection than ConUCB. Specifically, the improvements of total running time over ConUCB are 72.32% for ConLinUCB-BS and 57.32% for ConLinUCB-MCR. The main reason is that our algorithms estimate the unknown user preference vector in one single step, whereas ConUCB does it in two separate steps as mentioned before. For ConLinUCB-BS, the time costed in the key-term selection is almost negligible, since it just randomly chooses a key-term from the precomputed barycentric spanner whenever a conversation is allowed.

7 Conclusion

In this paper, we introduce ConLinUCB, a general framework for conversational bandits with efficient information incorporation. Based on this framework, we propose ConLinUCB-BS and ConLinUCB-MCR, with explorative key-term selection strategies that can quickly elicit the user’s potential interests. We prove tight regret bounds of our algorithms. Particularly, ConLinUCB-BS achieves a bound of $O(d\sqrt{T\log T})$, much better than $O(d\sqrt{T}\log T)$ of the classic ConUCB. In the empirical evaluations, our algorithms dramatically outperform the classic ConUCB. For future work, it would be interesting to consider the settings with knowledge graphs (Zhao et al. 2022), hierarchy item trees (Song et al. 2022), relative feedback (Xie et al. 2021) or different feedback selection strategies (Letard et al. 2020, 2022), and use our framework and principles to improve the performance of existing algorithms.

8 Acknowledgement

The corresponding author Shuai Li is supported by National Natural Science Foundation of China (62006151) and Shanghai Sailing Program. The work of John C.S. Lui was supported in part by the RGC’s GRF 14215722.

Appendix A Proof of Lemma 1

**Proof. ** According to the closed-form solution of $\bm{\theta}_{t}$ in Eq. (5) (6), we can calculate the estimation error as follows

[TABLE]

We can then bound the projection of the estimation error onto the direction of the action vector $\bm{x}_{a}$:

[TABLE]

where Eq. (12) is by the Cauchy–Schwarz inequality, Eq. (13) is by the inequality of the matrix operator norm, and Eq. (14) is because $\lambda_{min}(\bm{M}_{t})\geq\beta,\,\,\left\|\bm{M}_{t}^{-\frac{1}{2}}\right\|_{2}=\sqrt{\lambda_{max}(\bm{M}_{t}^{-1})}=\sqrt{\frac{1}{\lambda_{min}(\bm{M}_{t})}}\leq\sqrt{\frac{1}{\beta}}$.

Theorem 1 in (Abbasi-Yadkori, Pál, and Szepesvári 2011) suggests that with probability at least $1-\delta$

[TABLE]

where $det(\cdot)$ denotes the determinate of the argument.

We have

[TABLE]

where $\lambda_{i},i=1,2,\ldots,d$ denotes the eigenvalues of the matrix $\bm{M}_{t}$, $trace(\bm{M}_{t})$ denotes the trace of $\bm{M}_{t}$, Eq. (16) follows by the inequality of arithmetic and geometric means, Eq. (17) follows since the trace of a matrix is equal to the sum of its eigenvalues.

Plugging the above inequality and $det(\beta\bm{I})=\beta^{d}$ into Eq. (15), we can get

[TABLE]

The result then follows by plugging Eq. (18) into Eq. (14), and the fact that $\left\|\bm{\theta}^{*}\right\|_{2}\leq 1$. $\Box$

Appendix B Proof of Lemma 2

**Proof. ** Recall that in ConLinUCB-BS, the key-terms are uniformly sampled from the pre-computed barycentric spanner $\mathcal{B}$, i.e., ${k}\sim\text{unif}(\mathcal{B})$. Therefore we have

[TABLE]

Using Eq. (11) in the Lemma 7 in (Li and Zhang 2018), and the fact that $b_{t}=b\cdot t$, then with probability at least $1-\delta$ for $\delta\in(0,\frac{1}{8}]$, we have

[TABLE]

for all $t\geq t_{0}=\frac{256}{b\lambda_{\mathcal{B}}^{2}}\log(\frac{128d}{\lambda_{\mathcal{B}}^{2}\delta})$.

Then, by Courant–Fischer theorem (Ikebe, Inagaki, and Miyamoto 1987), the fact that $\left\|\bm{x}_{a}\right\|_{2}=1$, together with Eq. (20), we have that for any $t\geq t_{0}$,with probability at least $1-\delta$ for $\delta\in(0,\frac{1}{8}]$,

[TABLE]

$\Box$

Appendix C Proof of Theorem 3

**Proof. ** We denote the instantaneous regret at round $t$ as $R_{t}$. With the definition of the cumulative regret given in Eq. (2), the arm selection strategy shown in Eq. (7) and Lemma 1, we can bound the regret $R_{t}$ at each round $t=1,2,3,4,...,T$ as follows

[TABLE]

With Lemma 2, together with the assumption that $r_{t}\leq 1$ for any $t$, with probability at least $1-\delta$ for some $\delta\in(0,\frac{1}{4}]$, we can get

[TABLE]

where Eq. (22) follows since $\alpha_{t}$ is non-decreasing in $t$.

The result follows by plugging in the definition of $t_{0}$ and $\alpha_{T}$. $\Box$

Appendix D Proof of Theorem 4

**Proof. ** We first prove the following result:

For any two positive definite matrices $\bm{A},\bm{B}\in\mathbb{R}^{d\times d}$, and any vector $\bm{x}\in\mathbb{R}^{d}$, we have:

[TABLE]

This result can be proved by the following arguments:

[TABLE]

where Eq. (24) follows from the Woodbury matrix identity (Woodbury 1950), and Eq. (25) is because $(\bm{B}^{-1}+\bm{A}^{-1})^{-1}$ is a positive definite matrix.

With the above result, then following Eq. (21) and the Cauchy–Schwarz inequality, we can get

[TABLE]

Using Lemma 11 in (Abbasi-Yadkori, Pál, and Szepesvári 2011), with probability at least $1-\delta$, we can get

[TABLE]

Following similar steps as in Eq. (17), we can get that

[TABLE]

Therefore we have

[TABLE]

The result then follows by plugging in the definition of $\alpha_{T}$ and Eq. (29) into Eq. (26). $\Box$