Preselection Bandits

TL;DR

This paper introduces the Preselection Bandit problem where a learner preselects subsets of options to a user, learns preferences from stochastic choices modeled by Plackett-Luce, and develops algorithms with near-optimal regret bounds.

Contribution

It formalizes the Preselection Bandit problem, introduces a regret measure, derives lower bounds, and proposes algorithms with matching upper bounds up to logarithmic factors.

Findings

Derived lower bounds on expected regret.

Proposed algorithms achieve near-optimal regret bounds.

Modeling user choices with Plackett-Luce enhances learning efficiency.

Abstract

In this paper, we introduce the Preselection Bandit problem, in which the learner preselects a subset of arms (choice alternatives) for a user, which then chooses the final arm from this subset. The learner is not aware of the user's preferences, but can learn them from observed choices. In our concrete setting, we allow these choices to be stochastic and model the user's actions by means of the Plackett-Luce model. The learner's main task is to preselect subsets that eventually lead to highly preferred choices. To formalize this goal, we introduce a reasonable notion of regret and derive lower bounds on the expected regret. Moreover, we propose algorithms for which the upper bound on expected regret matches the lower bound up to a logarithmic term of the time horizon.

Tables

Table 1. Table 1 : Problem instances with different optimal subsets (indicated in bold font) for the regret in ( 8 ) with n = 5 𝑛 5 n=5 and l = 3 𝑙 3 l=3 (omitted subsets had smaller utilities throughout).

S	{1, 2, 3}	{1, 2, 5}	{1, 3, 5}	{1, 4, 5}	{2, 3, 4}
$v=(1,0.122,0.044,0.037,0.017),\gamma =1$
$U\left(S\right)$	0.872	0.891	0.945	0.951	0.0896
$v=(1,0.681,0.572,0.543,0.399),\gamma =1$
$U\left(S\right)$	0.795	0.780	0.754	0.749	0.604
$v=(1,0.681,0.572,0.543,0.171),\gamma =1$
$U\left(S\right)$	0.795	0.806	0.778	0.773	0.604

Table 2. Table 2 : Problem instances with different optimal subsets (indicated in bold font) for the regret in ( 8 ) with n = 5 𝑛 5 n=5 and l = 3 𝑙 3 l=3 (omitted subsets had smaller utilities throughout).

S	{1, 2, 3}	{1, 2, 4}	{1, 2, 5}	{1, 3, 5}	{2, 3, 4}
$v=(1,0.681,0.572,0.543,0.171),\gamma =1$
$U\left(S\right)$	0.795	0.791	0.806	0.778	0.605
$v=(1,0.681,0.572,0.543,0.171),\gamma =20$
$U\left(S\right)$	1.000	1.000	1.000	1.000	0.676
$v=(1,0.681,0.572,0.543,0.171),\gamma =0.05$
$U\left(S\right)$	0.753	0.744	0.630	0.593	0.599

Table 3. Table 3: Empirical standard deviations of the cumulative regret for the different time horizon steps for the scenarios ( n , l ) = ( 20 , 4 ) 𝑛 𝑙 20 4 (n,l)=(20,4) and ( n , l ) = ( 30 , 5 ) . 𝑛 𝑙 30 5 (n,l)=(30,5).

	$(n,l)=(20,4)$
$T$	2000	4000	6000	8000	10000
TRCB	21.47	32.93	43.73	57.51	66.36
UCB-Oracle	43.90	79.27	119.19	165.42	202.54
UCB-Sampling	98.31	187.52	280.59	370.10	479.20
TS-Oracle	7.74	10.01	11.37	13.42	14.06
TS-Sampling	86.11	161.01	235.16	329.54	429.52
TS-Oracle-Corr	21.84	43.65	63.73	87.99	111.97
	$(n,l)=(30,5)$
TRCB	20.54	34.81	40.84	54.91	58.88
UCB-Oracle	34.45	65.40	102.52	143.31	172.84
UCB-Sampling	75.21	150.24	225.29	311.10	385.72
TS-Oracle	6.91	9.48	11.17	13.43	13.91
TS-Sampling	53.66	101.35	175.86	246.12	284.47
TS-Oracle-Corr	18.01	38.74	55.92	80.59	97.91

Equations

$$q_{i_1,\dots ,i_l}\text{vbox}..={\sum }_{\mathbf{r}\in {\mathbb{S}}_n:\mathbf{r}(i_1)=\min \left(\mathbf{r}(i_1),\dots ,\mathbf{r}(i_l)\right)}\mathbb{P}(\mathbf{r}).$$

$${\mathbb{P}}_{\theta }(\mathbf{r})={\prod }_{i=1}^n\frac{{\theta }_{{\mathbf{r}}^{-1}(i)}}{{\theta }_{{\mathbf{r}}^{-1}(i)}+\dots +{\theta }_{{\mathbf{r}}^{-1}(n)}},$$

$$q_{i_1,\dots ,i_l}=\frac{{\theta }_{i_1}}{{\theta }_{i_1}+{\theta }_{i_2}+\dots +{\theta }_{i_l}}$$

$$O_{i,j}\text{vbox}..=\frac{{\theta }_i}{{\theta }_j},i,j\in [n],$$

$$O_{i,j}=\frac{{\theta }_i}{{\theta }_j}=\frac{{\theta }_i}{{\theta }_j}\cdot \frac{{\sum }_{t\in S_{i,j}}{\theta }_t}{{\sum }_{t\in S_{i,j}}{\theta }_t}=\frac{q_{i,S_{-i,j}}}{q_{j,S_{i,-j}}}.$$

$$\displaystyle\Theta=\Big{\{}\theta=(\theta_{1},\ldots,\theta_{n})^{T}\in$$

$$\displaystyle\theta_{max}\mathrel{\vbox{\hbox{\footnotesize.}\hbox{\footnotesize.}}}=\max_{i}\theta_{i}=1\Big{\}},$$

$${\theta }_i=v_i^{\gamma },\forall i\in [n],$$

$$\begin{array}{rl}\mathrm{U}(S):=\mathrm{U}(S;v,\gamma )& =\sum _{i\in S}{v}_i\cdot q_{i,S\backslash \{i\}}\\ & =\frac{{\sum }_{i\in S}{v}_i^{1+\gamma }}{{\sum }_{i\in S}{v}_i^{\gamma }}.\end{array}$$

$$S^{\ast }\in \{\begin{array}{ll}{\arg \max }_{S\subseteq [n],|S|=l}\mathrm{U}(S),& \text{mbox}if\mathbb{A}={\mathbb{A}}_l,\\ {\arg \max }_i{\theta }_i,& \text{mbox}if\mathbb{A}={\mathbb{A}}_{full}.\end{array}$$

$$r(S):=\mathrm{U}(S^{\ast })-\mathrm{U}(S),S\in \mathbb{A}.$$

$$\displaystyle\mathcal{R}(T):=\sum\nolimits_{t=1}^{T}r(S_{t})=\sum\nolimits_{t=1}^{T}\big{(}\mathrm{U}(S^{*})-\mathrm{U}(S_{t})\big{)}.$$

$$\displaystyle\mathbb{E}_{\theta}\big{(}\mathcal{R}(T)\big{)}$$

$$\ge \min \{1,1/\gamma \}C\sqrt{nT}$$

$$\sup\nolimits_{\theta\in\Theta}\mathbb{E}_{\theta}\big{(}\mathcal{R}(T)\big{)}\geq C\,\min\{1,1/\gamma\}\,\sqrt{T}\,.$$

$$\displaystyle\sup\nolimits_{\theta\in\Theta}\Big{(}\min_{i\notin S^{*}}\,($$

$$\ge C\min \{1,1/\gamma \}(n-1)\log (T).$$

$$\tilde{\mathrm{U}}(S)=\tilde{\mathrm{U}}(S;O_J,\gamma )\text{vbox}..=\frac{{\sum }_{i\in S}{O}_{i,J}^{\text{nicefrac}(1+\gamma )\gamma }}{{\sum }_{i\in S}{O}_{i,J}},$$

$${\sup }_{\theta \in \Theta }{\mathbb{E}}_{\theta }^{\mathrm{TRCB}}\mathcal{R}(T)$$

$$\le C\frac{\max \{{\theta }_{min}^{(\gamma -1)/(\gamma )},{\theta }_{min}^{(1-\gamma )/(\gamma )}\}}{\gamma {\theta }_{min}^{2(3+\gamma )}}\sqrt{nT\log (T)},$$

$${\sup }_{\theta \in \Theta }$$

$$+C_1\frac{\max \{{\theta }_{min}^{(\gamma -1)/(\gamma )},{\theta }_{min}^{(1-\gamma )/(\gamma )}\}}{\gamma {\theta }_{min}}\times$$

$$\sum _{i\in [n]\backslash S^{\ast }}\frac{\log (T){\sum }_{i\in [n]\backslash S^{\ast }}({\theta }_{max}-{\theta }_i)}{({\theta }_{max}-{\theta }_i{)}^2}$$

$$\mathrm{KL}\big{(}P,\,Q\big{)}\leq\sum_{i=1}^{m}\frac{\varepsilon_{i}^{2}}{q_{i}}.$$

$$1-(1-\delta {)}^{1/\gamma }\ge \min \{1,1/\gamma \}\delta .$$

$$f^{\prime }(x)$$

$$f^{\prime \prime }(x)$$

$${\theta }_S(i):=\{\begin{array}{ll}1,& i\in S,\\ 1-\epsilon ,& i\notin S,\end{array}$$

$$\begin{array}{rl}\mathrm{U}(S;v_S,\gamma )-\mathrm{U}(\tilde{S};v_S,\gamma )\ge 1-\frac{(l-1)+(1-\epsilon {)}^{\frac{(1+\gamma )}{\gamma }}}{l-\epsilon }& =\frac{(1-\epsilon )(1-(1-\epsilon {)}^{\frac{1}{\gamma }})}{l-\epsilon }>\frac{\min \{1,1/\gamma \}\epsilon }{2l},\end{array}$$

$$\begin{array}{rl}{\mathbb{E}}_S\sum _{t=1}^T\mathrm{U}(S,{\theta }_S)-\mathrm{U}(S_t^{\phi },{\theta }_S)& \ge \frac{\min \{1,1/\gamma \}\epsilon }{2l}\sum _{i\notin S}{\mathbb{E}}_S{N}_i.\end{array}$$

$$\underset{\theta \in \Theta }{\sup }{\mathbb{E}}_{\theta }\mathcal{R}(T)$$

$$\ge \frac{1}{\left(\genfrac{}{}{0ex}{}{n}{l}\right)}\sum _{S\in {\mathbb{A}}_l}{\mathbb{E}}_S\sum _{t=1}^T\mathrm{U}(S,{\theta }_S)-\mathrm{U}(S_t^{\phi },{\theta }_S)$$

$$\ge \frac{1}{\left(\genfrac{}{}{0ex}{}{n}{l}\right)}\sum _{S\in {\mathbb{A}}_l}\sum _{i\notin S}\frac{\min \{1,1/\gamma \}\epsilon }{2l}{\mathbb{E}}_S{N}_i$$

$$\displaystyle=\frac{\min\{1,1/\gamma\}\,\varepsilon}{2}\,\Big{(}T-\frac{1}{l\,\binom{n}{l}}\sum_{S\in\mathbb{A}_{l}}\sum_{i\in S}\,\mathbb{E}_{S}N_{i}\Big{)},$$

$$\displaystyle\sup_{\theta\in\Theta}\mathbb{E}_{\theta}\mathcal{R}(T)\geq\frac{\min\{1,1/\gamma\}\,\varepsilon\,T}{2}\Big{(}\frac{2}{3}-\sup_{S^{\prime}\in\mathbb{A}_{l-1}}\,\sqrt{\sum_{i\in S^{\prime}}\frac{\mathrm{KL}\big{(}\mathbb{P}_{S^{\prime}},\mathbb{P}_{S^{\prime}\cup\{i\}}\big{)}}{2(n-l+1)}}\ \Big{)}.$$

Full text

Preference Learning, Exploration, Exploitation, Online Learning, Plackett-Luce, MNL model

1 Introduction

The setting of preference-based multi-armed bandits or dueling bandits (Busa-Fekete et al., 2018) is a generalization of the standard stochastic multi-armed bandit (MAB) problem (Lattimore & Szepesvári, 2020). Instead of assuming numerical rewards of individual arms (choice alternatives), the former is based on pairwise preferences between arms. In this paper, we introduce the Preselection Bandit (or simply Pre-Bandit) problem, which is closely related to the preference-based setting, especially to the recent variant of battling bandits (Saha & Gopalan, 2018).

Our setting involves an agent (learner), which preselects a subset of arms, and a selector (a human user or another algorithm), which then chooses the final arm from this subset. This setting is motivated by various practical applications. In information retrieval, for example, the role of the agent is played by a search engine, and the selector is the user who seeks a certain information. Another example is online advertising, where advertisements recommended to users can be seen as a preselection. As a concrete application, we are currently working on the problem of algorithm (pre-) selection (Kerschke et al., 2019), where the (presumably) best-performing algorithm needs to be chosen from a pool of candidates.

In the beginning, the agent is not aware of the selector’s preferences. However, the choices made by the latter reveal information about these preferences, from which the agent can learn. Due to time constraints, information asymmetry, or other reasons, we do not assume the selector to act perfectly, which means that it may miss the actually best among the preselected arms. In web search, for example, a user clicks on links based on limited information such as snippets, but without knowing the full content behind. Likewise, in algorithm selection, the final choice might be made on the basis of a cross-validation study, i.e., estimated performances that not guarantee the identification of the truly best algorithm. By modeling the selector’s actions by means of the Plackett-Luce (PL) model (Luce, 1959; Plackett, 1975), we allow some randomness in the process of decision making. The agent’s main task is to preselect subsets that eventually lead to highly preferred choices. To formalize this goal, we introduce a reasonable notion of regret based on utilities of preselected subsets, where an arm’s (latent) utility is weighted with the probability of choosing this arm from the subset. In particular, this allows for capturing decision-making biases of users as studied intensively in behavioral economics or psychology.

We study two variants of the problem. In the first variant, which we call restricted Pre-Bandit problem, the size of the preselection is predefined and fixed throughout. In the second variant, the flexible Pre-Bandit problem, the agent is allowed to adjust the size of the preselection in every round. For these settings, we derive lower bounds on the expected regret. Moreover, for both scenarios, we propose active learning algorithms for which the upper bound on expected regret matches the lower bound (possibly) up to a logarithmic term of the time horizon.

We discuss related work in Section 2. In Section 3, we introduce the notation used throughout the paper, and also give a concise review of the PL model and some of its properties. In Section 4, the Pre-Bandit problem is formally introduced, together with a reasonable notion of regret, for which lower bounds with respect to the time horizon are verified. Near-optimal algorithms for the two variants of the Pre-Bandit problem are provided in Section 5. We devote Section 6 to a simulation study demonstrating the usefulness and efficiency of our algorithms. Finally, Section 7 summarizes our results and discusses directions for future work. All proofs of the theoretical results are deferred to the supplements.

2 Related Work

Bandit problems with the possibility of using more than one arm at a time have been considered in various ways in the literature. However, as will be detailed in the following, none of the previous works encompasses the problem considered in this paper. Due to the specific meaning of the subset choice as a preselection in practical applications, this also justifies a new name for the setting.

A variant of the MAB problem is the combinatorial bandit problem (Cesa-Bianchi & Lugosi, 2012; Kveton et al., 2015), in which the learner chooses a subset of arms in each time step, and then observes quantitative feedback, either in the form of rewards of each single arm (semi-bandit feedback) or the total sum of the rewards (bandit feedback) for the arms in the chosen subset. This differs fundamentally from our setting, in which no quantitative feedback is ever observed; instead, only qualitative feedback is provided, i.e., which arm is picked in a subset.

Qualitative forms of feedback for multiple arm choices at a time is considered in the realm of dueling, multi-dueling (Sui et al., 2017; Busa-Fekete et al., 2018) or battling bandits (Saha & Gopalan, 2018). The flexible Pre-Bandit problem has obvious connections to the latter settings, with the freedom of adjusting the size of comparison for each time instance and can be interpreted as a combinatorial bandit problem with qualitative feedback. Saha & Gopalan (2019a) investigate the effect of this flexibility in an active PAC-framework for finding the best arm under the PL model, while the active top-k-arm identification problem in this model is studied by Chen et al. (2018a). Recently, this scenario was considered in terms of a regret minimization problem with top-$m$-ranking feedback by Saha & Gopalan (2019b) for a straightforward extension of the dueling bandit notion of regret and not regarding the “value” of a subset in its entirety as we do (cf. Section 4.1).

Moreover, the algorithms suggested in Saha & Gopalan (2018) are not applicable within our scenario for the restricted Pre-Bandit problem, as either the algorithms are focusing on the linear-subset choice model, which is fundamentally different from our choice model (cf. Section 3) or the algorithms allow replicates of the same arm within a chosen subset, which is inadmissible within our scenario (and in many practical applications as well).

The Pre-Bandit problem also reveals parallels to the Dynamic Assortment Selection (DAS) problem (Caro & Gallien, 2007), where a retailer seeks to find an optimal subset of his/her available items (or products) in an online manner, so as to maximize the expected revenue (or equivalently minimize the expected regret). The DAS problem under the multinomial logit model, which is also known as the MNL-Bandits problem (Rusmevichientong et al., 2010; Sauré & Zeevi, 2013; Agrawal et al., 2016, 2017; Wang et al., 2018; Chen et al., 2018b) is especially close to our framework, as the corresponding concept of regret shares similarities with our definition of regret.

However, our problem can rather be seen as complementary, since we do not assume a priori known revenues for each item or any revenues at all. While this might be natural for the retail management problem, it is arguably less so for applications we have in mind, such as recommendation systems or algorithm (pre-)selection. In addition, we introduce a parameter in our setting that allows the learner to adjust its preselections with respect to the preciseness of the user’s selections. This might also be an interesting direction for future work for the DAS problem. Finally, to demonstrate the inappropriateness of the DAS algorithms for the restricted Pre-Bandit problem, we employ some of the algorithms in our experimental study.

Another quite related branch of research is the so-called stochastic click model (Zoghi et al., 2017; Lattimore et al., 2018), where a list of $l$ items is presented to the selector in each iteration. Scanning the list from the top to the bottom, there is a certain probability that the selector chooses the item at the current position, or otherwise continues searching (eventually perhaps not choosing any item). Thus, in contrast to our setting, the explicit order of the arms within a subset (or list) is a relevant aspect. Further, the resulting learning task boils down to finding the $l$ most attractive items, as these provably constitute the optimal list in this scenario (which is not necessarily the case for our setting).

3 Preliminaries

3.1 Basic Setting and Notation

We formalize our problem in the setting of preference-based multi-armed bandits (Busa-Fekete et al., 2018), which proceeds from a set of $n$ arms, each of which is considered as a choice alternative (item, option). We identify the arms by the index set $[n]\mathrel{\vbox{\hbox{\footnotesize.}\hbox{\footnotesize.}}}=\{1,\ldots,n\}$, where $n\in\mathbb{N}$ is arbitrary but fixed. Moreover, we assume a total preference order $\succ$, where $i\succ j$ means that the $i$th is preferred to the $j$th arm.

Let $\mathbb{A}_{l}$ be the set of all $l$-sized subsets of $[n]$ and $\mathbb{A}_{full}\mathrel{\vbox{\hbox{\footnotesize.}\hbox{\footnotesize.}}}=\cup_{l=1}^{n}\mathbb{A}_{l}$. Moreover, let $\mathbb{S}_{n}$ be the symmetric group on $[n]$, the elements of which we refer to as rankings: each $\mathbf{r}\in\mathbb{S}_{n}$ defines a ranking in the form of a total order of the arms $[n]$, with $\mathbf{r}(i)$ the position of arm $i$. We assume that $\mathbb{S}_{n}$ is equipped with a probability distribution $\mathbb{P}:\,\mathbb{S}_{n}\to[0,1]$. For an integer $l>1$ and a set of arms $\{{i_{1}},{i_{2}},\ldots,{i_{l}}\}\subseteq[n]$, the probability that $i_{1}$ is the most preferred among this set is given by

[TABLE]

3.2 The Plackett-Luce Model

The Plackett-Luce (PL) model (Plackett, 1975; Luce, 1959) is a parametric distribution on the symmetric group $\mathbb{S}_{n}$ with parameter $\theta=(\theta_{1},\ldots,\theta_{n})^{T}\in\mathbb{R}_{+}^{n},$ where each component $\theta_{k}$ corresponds to the strength of an arm $k,$ which we will refer to as score parameter. The probability of a ranking $\mathbf{r}\in\mathbb{S}_{n}$ under the PL model is

[TABLE]

where $\mathbf{r}^{-1}(i)$ denotes the index of the arm on position $i$. According to (2), PL models a stage-wise construction of a ranking, where in each round, the item to be put on the next position is chosen with a probability proportional to its strength. As a model of discrete choice, the PL distribution has a strong theoretical motivation. For example, it is the only model that satisfies the Luce axiom of choice (Luce, 1959), including independence from irrelevant alternatives (ILA property, see (Alvo & Yu, 2014)). Besides, it has a number of appealing mathematical properties. For instance, there is a simple expression for the $l$-wise marginals in (1):

[TABLE]

This probability is identical to the popular Multinomial Logit (MNL) model, which is a discrete choice probability model considered in various frameworks (Train, 2009). For our purposes, the use of the relative scores

[TABLE]

will turn out to be advantageous, as they are directly affected by the ILA property of the PL model. Indeed, for $i,j\in[n]$, let $S_{i,j}\in\mathbb{A}_{l}$ be such that $i,j\in S_{i,j}.$ Furthermore, define $S_{-i,j}\mathrel{\vbox{\hbox{\footnotesize.}\hbox{\footnotesize.}}}=S_{i,j}\backslash\{i\}$ and similarly $S_{i,-j}\mathrel{\vbox{\hbox{\footnotesize.}\hbox{\footnotesize.}}}=S_{i,j}\backslash\{j\}$ for $i,j\in[n].$ Then, for any such a set $S_{i,j}$, (3) and (4) imply

[TABLE]

Without restricting the parameter space $\Theta=\{\theta\in\mathbb{R}_{+}^{n}\}$, the PL model in (2) is not (statistically) identifiable, as $\theta\in\mathbb{R}_{+}^{n}$ and $\tilde{\theta}=C\,\theta$ for any constant $C>0$ lead to the same models, i.e. $\mathbb{P}_{\theta}=\mathbb{P}_{\tilde{\theta}}.$ Restricting the parameter space by assuming some normalization condition on the score parameters fixes this issue. Thus, we consider as parameter space the (restricted) unit square w.r.t. the infinity norm,

[TABLE]

which leads to an identifiable statistical model $(\mathbb{P}_{\theta})_{\theta\in\Theta}$ and naturally yields a normalization of each individual score parameter easing the fast grading of an arm’s utility.

For technical reasons, we additionally exclude models that allow scores below a certain threshold $\theta_{min}$ (which will be a small constant), as the relative scores in (4) are then well-defined for any pair $(i,j)\in[n]^{2}.$

3.3 Degree of Preciseness

In our setting, we model the score parameter $\theta$ as

[TABLE]

where $v_{i}\in\mathbb{R}_{+}$ represents the (latent) utility of arm $i,$ while $\gamma\in(0,\infty)$ represents the degree of preciseness of the user’s selections: The higher $\gamma,$ the more (2) resembles a point-mass distribution on the ranking modeling a precise selector that is always able to identify the best arm, while the lower $\gamma$, the more (2) resembles a uniform distribution modeling a selector acting purely at random. The effect of $\gamma$ on the $l$-marginals in (3) is quite similar.

Note that $v_{1},\ldots,v_{n},\gamma$ are not separately identifiable (c.f. Section 3.2 in (Train, 2009)), but $\theta_{1},\ldots,\theta_{n}$ are identifiable under our assumptions on the parameter space $\Theta$ above. Hence, by fixing $\gamma$, the latent utilities $v_{1},\ldots,v_{n}$ are guaranteed to be identifiable.

4 The Pre-Bandit Problem

The considered online learning problem proceeds over a finite time horizon $T$. For each time instance $t\in[T]$, the agent (i.e., the learner) suggests a subset $S_{t}\in\mathbb{A}$, where $\mathbb{A}$ is the action space. The agent’s action $S_{t}$ is based on its observations so far. As a new piece of information, it observes the selector’s choice (i.e., the user or the environment) of an arm $i_{t}$ among the offered subset $S_{t}$ (with probability $q_{i_{t},S_{t}\backslash\{i_{t}\}}$ given by (3)).

Suppose $r:\mathbb{A}\to\mathbb{R}_{+}$ is a suitable regret function (to be defined in the next section below). The goal of the learner resp. the agent is to preselect the available arms by means of subsets $S_{t}$ in every time instance $t$ such that the expected cumulative regret over the time horizon, that is $\mathbb{E}_{\theta}\sum_{t=1}^{T}r(S_{t})$ with $\theta\in\Theta$, is minimized. The problem is analyzed for two possible characteristics of the action space:

• •

(Restricted Preselection) $\mathbb{A}=\mathbb{A}_{l},$ i.e., a preselection consists of exactly $l$ many arms, where $l$ is a fixed integer strictly greater than one.

• •

(Flexible Preselection) $\mathbb{A}=\mathbb{A}_{full},$ i.e., a preselection can be any non-empty subset of $[n].$

In the following, we introduce sensible notions of regret for the considered problem setting. The key question we then address is the following: What is a good preselection to present the selector? Moreover, we provide a lower bound on the related expected cumulative regret.

4.1 Regret Definition

Assuming the selector to behave according to the PL model with score parameter (5), the expected utility of suggesting $S$ is given by

[TABLE]

Indeed, if $i_{t}\in[n]$ is the chosen arm at time $t,$ then

\mathbb{E}(v_{i_{t}}\,|\,S)=\sum_{i\in S}v_{i}\cdot\mathbb{P}(\mbox{i is chosen}\,|\,S)=\mathrm{U}(S).

Hence, the corresponding optimal preselection is

[TABLE]

The (instantaneous) regret suffered by the selector is anticipated by the agent through

[TABLE]

Thus, if $S_{1},\ldots,S_{T}$ are suggested for times $1$ to $T$, respectively, the corresponding cumulative regret over $T$ is

[TABLE]

Remark 1* (Relations to dueling bandits and battling bandits).*

Note that the optimal subset for $\mathbb{A}=\mathbb{A}_{full},$ i.e., for the flexible Pre-Bandit problem, always consists of the items whose score parameters equal the overall highest score $\theta_{max}.$ Thus, like for the dueling bandits and battling bandits problem, the goal is to find the best arm(s). However, whilst in the latter settings only pairwise resp. fixed $l$-wise comparisons of arms are observed, we allow to draw comparisons of arbitrary size. In addition, the restricted Pre-Bandit problem can be interpreted as a dueling resp. battling bandit problem. Compared to the latter, however, the notion of regret has a more natural meaning in our setting. This is due to the different semantics of a selection of a pair (or any subset) of arms, which is a preselection that eventually leads to a concrete choice. In other words, the regret in (8) focuses on the perceived preference of a subset by regarding the “value” of a subset in its entirety.

Remark 2* (No-choice option).*

In the related branches of literature (cf. Section 2), it is common to assume an additional choice alternative which represents the possibility of the user choosing none of the alternatives in the preselection. Formally, this can be expressed in our setting by extending the $n$-dimensional parameter space $\Theta$ to $n+1$ dimensions by augmenting it with a dummy score parameter $\theta_{0}\in(\theta_{min},1]$ representative for this no-choice option. As this option is always available, this dummy item is part of every preselection, and consequently its latent utility affects only the choice probabilities in (6). However, although we refrain from incorporating the no-choice alternative in this paper, it is straightforward to show similar lower bounds as below for this problem and to modify the suggested algorithms to this scenario.

4.2 Most preferred Subsets

One tempting question is how the most preferred subsets look like, given our definition of regret. As already mentioned, the optimal preselection $S^{*}$ for the flexible Pre-Bandit variant consists of the items with the same highest score parameter. However, in the restricted Pre-Bandit variant, the optimal preselection does not necessarily consist of the $l$ items with the highest scores in general, as the following examples demonstrate.

Example 1**.**

In Table 1, we provide three problem instances with fixed degree of preciseness $\gamma=1$ for $n=5$ and the corresponding expected scores of (the relevant) 3-sized subsets of $[n]$. In the first instance, where one arm has a much higher utility than the remaining ones, it is favorable to suggest this high utility arm together with the arms having smallest utility. This is due to the large differences between the utilities, so that the selector will take the best arm with a sufficiently high probability. Roughly speaking, the best strategy for the agent is to make the problem for the selector as easy as possible.

The second instance is different, as the optimal preselection for the agent now consists of the top-3 arms with the highest scores. This comes with a non-negligible probability of missing the optimal arm, however, since the runner-up arms are sufficiently strong, the regret can be tolerated. On the other hand, adding a poor arm would be suboptimal, as one cannot be certain enough that it will not be taken. But by reducing the score for the worst arm notably as in the third instance, the worst arm substitutes the third best, as then the best item can again be better distinguished from the suboptimal ones inside the optimal subset.

As suggested by this example, a reasonable strategy is to compose the preselection of subsets of best and worst arms, respectively. In fact, we show in the supplementary material (Section D) that the optimal subsets for the restricted Pre-Bandit problem are always composed of best and worst arms with the overall best arm(s) mandatory inside the optimal subset.

The obvious rationale of adding a strong arm is to guarantee a reasonably high utility, whereas a poor arm merely serves as a decoy to increase the probability of choosing the best arm. Such effects are known in the literature on decision theory as the attraction effect or the decoy effect, see (Dimara et al., 2017) and references therein. In particular, our definition of regret is able to capture this effect and consequently emphasizes that our regret aims at penalizing difficult decisions for the selector in the restricted case.

However, the manifestation of the decoy effect and consequently its demand for the application at hand, can be steered by the learner through fixing the degree of preciseness $\gamma$ as illustrated by the next example.

Example 2**.**

In Table 2, we investigate the effect of the degree of preciseness $\gamma$ for the third instance of Example 1. In the first instance, where the degree of preciseness is moderate, i.e., $\gamma=1,$ the attraction effect is still present. But by increasing the degree of preciseness to $\gamma=20$ as in the second instance, the (rounded) utilities of all subsets containing the best item are the same. For such a problem instance, it is enough to provide the best item in the preselection, as the user’s probability of choosing the best item is sufficiently high in these cases and the best arm will not be missed. In the last instance, the utility monotonically decreases with the sum of scores of the subset $S$ if the degree of preciseness is reduced to $\gamma=0.05.$ This is due to a throughout non-negligible probability for choosing the worst item inside the preselected subset $S.$ Thus, in order to dampen this effect and the resulting regret, it is best to preselect the top arms.

In view of this example, the learner can interpolate between the two extreme cases of users by varying $\gamma:$ Taking a sufficiently large $\gamma,$ the learner can model a very precise user for whom it suffices to preselect a subset that just entails the best item. Using a sufficiently small $\gamma$ instead, the learner is able to reproduce a random user for whom it is best to compose the preselection of the best items.

Remark 3*.*

Note that, in terms of regret, the case of a very precise user is related to the weak regret of the dueling bandits problem (Yue et al., 2012; Chen & Frazier, 2017), where no regret occurs whenever the best arm is participating in the duel. Similarly, the semantics of the regret in the case of a random user corresponds to the strong regret of the dueling bandits problem, where the regret is zero only when the best arm is duelled with itself.

4.3 Lower Bounds

In this section, we prove lower bounds on the expected regret defined in (9) for the two types of Pre-Bandit problems.

Theorem 4.1**.**

[Restricted Preselection Bandits] Let $n\in\mathbb{N},$ $l\leq n/4$, and $T\geq n$ be integers. Then, for any algorithm $\varphi$ suggesting an $l$-sized subset $S_{t}^{\varphi}$ at time $t,$

[TABLE]

holds for any $\theta\in\Theta$ and any $\gamma\in(0,\infty),$ where $C>0$ is some constant independent of $n,l$, and $T.$

Remark 4*.*

The order of the lower bound in Theorem 4.1 coincides with the lower bound on the expected regret derived by Chen & Wang (2018) for the DAS problem under the MNL model with capacity constraints. In particular, the preselection size $l$ does not affect the order, at least if it is smaller than $n/4.$ Although the lower bounds are theoretically of the same order, it is not directly possible to use the lower bound results of Agrawal et al. (2016) or Chen & Wang (2018), as in both proofs the probability of the no-choice option is assumed to be strictly positive, and the revenues all equal 1. Moreover, the results for the stochastic click model are quite different from ours (cf. Theorem 2 by Lattimore et al. (2018)), as there the size of the subset $l$ is present in the lower bound. Therefore, we provide a proof in the supplementary material (Section A).

Theorem 4.2**.**

[Flexible Preselection Bandits] Let $n\in\mathbb{N}$ and $T\geq n$ be integers. Then, for any algorithm $\varphi$ suggesting subset $S_{t}^{\varphi}\in\mathbb{A}_{full}$ at time $t,$ the following holds for any $\gamma\in(0,\infty)$:

(i) [Gap-independent version] There exists a constant $C>0$ independent of $n$ and $T$, such that

[TABLE]

(ii) [Gap-dependent version] If $\varphi$ is a no-regret algorithm (cf. Definition 2 in (Saha & Gopalan, 2019b)), there exists a constant $C>0$ independent of $n$ and $T$, such that

[TABLE]

Remark 5*.*

Note that the gap-independent lower bound is independent of the number of arms $n.$ This is in line with the enhancement for the DAS problem for the uncapacitated compared to the capacitated MNL model (Wang et al., 2018). On the other hand, the gap-dependent lower bound depends on the number of arms $n$, and is of the same order as in the dueling bandit setting. In particular, compared to the dueling bandits setting, there is (theoretically) no improvement by offering subsets larger than two. This is in accordance with the observations made by Saha & Gopalan (2018, 2019b).

5 Algorithms

In this section, we propose the Thresholding-Random-Confidence-Bound (TRCB) algorithm stated in Algorithm 1. This algorithm returns subsets $S_{1},\ldots,S_{T}$ for the restricted Pre-Bandit problem. As will be shown, it has a satisfactory upper bound for the expected cumulative regret in (9). For the flexible Pre-Bandit problem, we further suggest the Confidence-Bound-Racing (CBR) algorithm as stated in Algorithm 2. It is inspired by the idea of racing algorithms, initially introduced by Maron & Moore (1997) to find the best model in the framework of model selection.

5.1 The TRCB Algorithm

First of all, note that an estimation of the score parameter $\theta$ is not necessary for the goal of regret minimization. Instead, a proper estimation of the relative scores in (4) is sufficient. Indeed, maximizing the expected utility (6) is equivalent to maximizing the expected utility with respect to some reference arm $J,$ that is

[TABLE]

where $O_{J}=(O_{1,J},\ldots,O_{n,J}),$ simply because $\widetilde{\mathrm{U}}(S)=v_{J}^{-1}\cdot\mathrm{U}(S).$

Thanks to Lemma 1 by Saha & Gopalan (2019a), one can derive appropriate confidence region bounds based on a similar exponential inequality for the relative score estimates, so that one might be tempted to use a UCB-like policy for the restricted Pre-Bandit problem. However, the main problem of such an approach is UCB’s principle of “optimism in the face of uncertainty”, which tends to exclude arms with low score from a preselection. As we have seen in Example 1, such arms could indeed be part of the optimal subset $S^{*}$ depending on the specific value of $\gamma.$

The core idea of the TRCB algorithm is to solve this issue with a certain portion of pessimism. Instead of using the upper confidence bound estimates for the relative scores, a random value inside the confidence region of the relative score estimate is drawn (lines 7–8), so that pessimistic guesses for the relative scores are considered as well, which in turn ensures sufficient exploration of the algorithm. This sampling idea can be interpreted as a frequentist statistical version of Thompson Sampling. To exclude inconsistencies with the score parameter space (cf. Section 3.2), these random confidence values are appropriately thresholded.

Until the a priori unknown time horizon is reached (lines 3, 4, 13), the TRCB algorithm repeatedly does the following. Primarily, the arm with the highest total number of wins for the pairwise comparisons is determined as the reference arm $J$ (line 5). Next, for every other arm, a random value inside its confidence region for its relative score with respect to the reference arm $J$ is drawn with uniform distribution and appropriately thresholded (lines 7–10). These thresholded random values correspond to the current belief on the actual relative scores with respect to $J$ and are used to determine the preselection with the highest utility in (10) (line 11). After offering this preselection to the selector and observing its choice (line 12), the pairwise winning counts are updated (by breaking down the $l$-wise comparison into pairwise comparisons) as well as the estimates for the relative scores (line 13).

The following theorem shows that the upper bound for the worst-case cumulative regret of the proposed TRCB algorithm matches the information-theoretic lower bound on the cumulative regret in Theorem 4.1 with regard to $n$ and $T$ up to a logarithmic term of $T$ (the proof is given in Section B of the supplement).

Theorem 5.1**.**

If $C_{shrink}\in(0,1/2),$ then for any $\gamma\in(0,\infty)$ and any $T>n,$

[TABLE]

where $C>0$ is some constant independent of $n,l,$ $T$ as well as $\theta_{min}$ and $\gamma.$

Remark 6*.*

The maximization over $\mathbb{A}_{l}$ in Algorithm 1 (line 11) can be realized by Algorithm 3 provided in the supplementary material. It keeps the computational cost low by exploiting structural properties of the utility function $\mathrm{U}$ and the most preferred subsets (see Section 4.2).

5.2 The CBR Algorithm

The CBR algorithm is structurally similar to the TRCB algorithm. However, it uses estimates of the pairwise winning probabilities and the corresponding confidence intervals instead of the relative scores.

In particular, the CBR algorithm maintains a pool of candidates $A\in[n]$ and admits an arm $i\in A$ to be part of the preselection with a certain probability determined by the rate of uncertainty that $i$ could beat the current arm $J$ with the most winning counts. This uncertainty is expressed through the ratio between the length of the confidence interval for $q_{i,J}$ (cf. the definition in (1)) exceeding $1/2$ and the overall confidence interval’s length. More specifically, if $[l_{i}(t),u_{i}(t)]$ is the confidence interval for $q_{i,J}$ in time instance $t,$ then arm $i$ is included into the preselection with probability \sigma\big{(}\nicefrac{{(u_{i}(t)-1/2)}}{{(u_{i}(t)-l_{i}(t))}}\big{)}, where $\sigma:\mathbb{R}\to[0,1]$ is a sigmoidal function, i.e., a surjective monotone function with $\sigma(1/2)=1/2$ and $\sigma(x)>0$ iff $x>0.$ Note that the degree of preciseness $\gamma$ can be taken into account by the learner through the shape of $\sigma.$

Hence, if the confidence interval lies mostly above $1/2,$ that is $l_{i}(t)\approx 1/2,$ the chance is high that this particular arm could possibly beat the current best arm and consequently has a large probability of being included in the preselection. In contrast, if the upper bound of the confidence interval is beneath $1/2,$ that is $u_{i}(t)\leq 1/2,$ the arm is discarded from the pool of candidates (lines 12–14), as one can be sure that this arm is already beaten by another.

At the beginning, the major part of the arms have a high chance to be part of the preselection, which however decreases over the course of time until finally the preselection consists of only the best arm(s). In the repetition phase, the preselection is successively built starting from the current arm with the most total number of wins for the pairwise comparisons and adding arms from the active set depending on the outcome of a Bernoulli experiment (lines 5–11), whose success probability depends on the length of the confidence interval (of the arm’s pairwise winning probability against $J$) above $1/2.$ After offering this preselection to the selector and observing its choice (line 16), the pairwise winning counts and estimates on the pairwise winning probabilities are updated (line 17).

We have the following theorem for the upper bound on the cumulative regret for CBR, which matches the information-theoretic gap-dependent lower bound on the cumulative regret in Theorem 4.2 (the proof is given in Section C of the supplement).

Theorem 5.2**.**

There are universal constants $C_{0},C_{1}>0,$ which do not dependent on $T$ or $n$, such that

[TABLE]

for any $T>n,$ $\gamma\in(0,\infty)$.

6 Experiments

In this section, we investigate the performance of TRCB (Algorithm 1) as well as CBR (Algorithm 2) on synthetic data for some specific scenarios, while providing further scenarios in the supplementary material.

6.1 Restricted Pre-Bandit Problem

First, we analyze the empirical regret growth with varying time horizon $T$ for the restricted Pre-Bandit problem. We consider the case $n=10,$ $l=3$, and time horizons $T\in\{i\cdot 2000\}_{i=1}^{5}.$ The degree of preciseness is $\gamma=1$ throughout, and the score parameters $\theta=(\theta_{i})_{i\in[n]}$ are drawn uniformly at random from the $n$-simplex, i.e., without a restriction on their minimal value and thus allowing $\theta_{min}$ to be infinitesimal. The left plot in Figure 1 provides the performance of our algorithms together with some algorithms for the DAS problem (see Section E in the supplement for more information on these).

For the algorithms of the DAS problem, the best arm is set to be the no-choice option, thereby putting (most of) them in the advantageous position of knowing a priori one element of the optimal subset. Nevertheless, only TS-Oracle, with the advantage of knowing the best arm a priori, is able to slightly outperform TRCB in this scenario, whereas all other algorithms are distinctly outperformed by TRCB.

To explain this observation, recall our remark on UCB-like strategies in Section 5.1. The UCB-based algorithms UCB-Oracle resp. UCB-Sampling as well as the UCB-like approximation of the variance of TS-Oracle-Corr tend to exclude arms with a low score from the suggested subset, even though they are contained in the optimal preselection. TS-Oracle and TS-Sampling, which do not use upper confidence bounds and include low score arms in the suggested subsets, are performing much better. The gap between these two TS algorithms shows how heavily the algorithms depend on the assumption that the no-choice option corresponds to the highest scored arm, since we designed TS-Sampling such that, in each run, it samples once the best-arm from the top three arms according to an MNL model.

In summary, this simulation confirms that the introduced (restricted) Pre-Bandit problem is indeed a new framework that differs from the DAS problem. A naïve application of existing methods for the DAS problem is not suitable for this kind of problem.

6.2 Flexible Pre-Bandit Problem

Next, we investigate the empirical regret growth with varying time horizon $T$ and varying numbers of arms $n$ for the flexible Pre-Bandit problem. In addition, we compared our algorithms with the Double Thompson Sampling (DTS) algorithm by Wu & Liu (2016), which is considered state-of-the-art for the dueling bandits problem with a small numbers of arms Sui et al. (2017).

In the right picture of Figure 1, the results are displayed for the CBR resp. DTS algorithm on 1000 repetitions, respectively, with $n\in\{5,10,15\},$ $T\in\{i\cdot 2000\}_{i=1}^{5}$, and $\sigma(x)=(1\wedge x)1_{[0,\infty)}(x).$ The score parameters are generated randomly as before. It is clearly recognizable that CBR distinctly outperforms DTS in all scenarios, indicating that offering larger subsets is at least experimentally beneficial to find the best arm more quickly.

7 Conclusion

In this paper, we have introduced the Pre-Bandit problem as a practically motivated and theoretically challenging variant of preference-based multi-armed bandits in a regret minimization setting. More specifically, we proposed two scenarios, one in which preselections are of fixed size and another one in which the size is under the control of the agent. For both scenarios, we derived lower bounds on the regret of algorithms solving these problems. Moreover, we proposed concrete algorithms and analyzed their performance theoretically and experimentally.

Our new framework suggests a multitude of conceivable paths for future work. Most naturally, it would be interesting to analyze the Pre-Bandit problem under different assumptions on the user’s choice behavior—despite being natural and theoretically justified (McFadden, 2001; Train, 2009), the assumption of the PL model is relatively strong, and the question is to what extent it could be relaxed. The main challenge surely lies in defining a sensible notion of regret, but an extension to the nested logit-model (Chen et al., 2018c) or considering contextual information (Chen et al., 2018b) seems to be possible. However, it is worth noting that our derived lower bounds on the regret based on expected utilities in the spirit of (6) even hold for more general choice models, such as generalized random utility models (Walker & Ben-Akiva, 2002; Train, 2009), which encompass the PL model.

Last but not least, like the related dynamic assortment selection problem studied in operational research, the motivation of our new framework stems from practical applications. Therefore, we are also interested in applying our algorithms to real-world problems, such as algorithm (pre-)selection already mentioned in the introduction. In particular, the realm of algorithm selection seems to be a canonical candidate for our setting, as the decisions made are based on the noisy performance values of the algorithms, which justify a stochastic modeling of the process.

Acknowledgments

The authors gratefully acknowledge financial support by the Germany Research Foundation (DFG). Moreover, the authors would like to thank the Paderborn Center for Parallel Computation (PC2) for the use of the OCuLUS cluster.

Appendix A Proofs of Theorems 4.1 and 4.2

For the proofs of Theorem 4.1 and Theorem 4.2 we need the following result on the Kullback-Leibler divergence of categorical probability distributions, which is Lemma 3 in Chen & Wang (2018). Throughout the proofs we let $\gamma\in(0,\infty)$ be some arbitrary degree of preciseness.

Lemma A.1**.**

Let $P\sim\mbox{Cat}(p_{1},\ldots,p_{m}),$ i.e. $P(i)=p_{i}$   for $i=1,\ldots,m$ and $\sum_{i=1}^{m}p_{i}=1,$ as well as $Q\sim\mbox{Cat}(q_{1},\ldots,q_{m}),$ such that $q_{i}=p_{i}+\varepsilon_{i}$ and $|\varepsilon_{i}|<1$ for any $i=1,\ldots,m.$ Then,

[TABLE]

Moreover, we will need the following auxiliary result for all lower bound results.

Lemma A.2**.**

For any $\delta\in(0,1)$ and any $\gamma\in(0,\infty)$ it holds that

[TABLE]

Proof.

First, consider the case $\gamma\in(0,1].$ Then the assertion follows immediately as the left-hand side of the inequality is monotonically decreasing with $\gamma$ and for $\gamma=1$ the inequality is valid.

Next, let us consider the case $\gamma\in(1,\infty).$ The assertion is equivalent to showing that $f(x)=1-x\delta-(1-\delta)^{x}$ is non-negative for $x\in(0,1).$ The first and second derivatives are respectively

[TABLE]

By straightforward computations it can be shown that $f$ has a global maximum on $(0,1)$ at $x_{max}=\frac{\log\left(\frac{-\delta}{\log(1-\delta)}\right)}{\log(1-\delta)}$ and $f$ is strictly increasing on $(0,x_{max})$ and strictly decreasing on $(x_{max},1).$ As $\lim\limits_{x\to 0}f(x)=\lim\limits_{x\to 1}f(x)=0,$ we can conclude the lemma. ∎

Proof of Theorem 4.1.

We will use a similar proof technique as in Chen & Wang (2018). Let $\varphi$ be some arbitrary algorithm suggesting the $l$-sized subsets (preselections) $(S_{t}^{\varphi})_{t\in[T]}\subset\mathbb{A}_{l}.$ For a set $S\in\mathbb{A}_{l}$ we write $\theta_{S}=(\theta_{S}(1),\ldots,\theta_{S}(n))$ to denote the score parameter of the PL-model with components given by

[TABLE]

where $\varepsilon\in(0,1/2)$ is some hardness parameter specified below. Note that for any $S\in\mathbb{A}_{l}$ the score parameter $\theta_{S}$ is an element of the parameter space $\Theta$. For sake of convenience, we will write $\mathbb{P}_{S}$ and $\mathbb{E}_{S}$ to express the law and expectation associated with the parameter $\theta_{S},$ i.e., $\mathbb{P}_{S}=\mathbb{P}_{\theta_{S}}.$ Recall the decomposition in (5) such that we have $\theta_{S}(i)=v_{S}(i)^{\gamma}$ for some suitable $v_{S}(i)$’s respectively and we define $v_{S}$ in the same spirit as $\theta_{S}.$

First, for any $S,\tilde{S}\in\mathbb{A}_{l}$ with $S\neq\tilde{S}$ it holds that

[TABLE]

where we used for the last step $1-\varepsilon\geq 1/2$ and $l-\varepsilon<l$ as well as Lemma A.2. For $i\in[n]$ let $N_{i}(t)=\sum_{s=1}^{t}1_{\{i\in S_{s}^{\varphi}\}}$ denote the number of times an arm $i$ is part of a preselection till time instance $t$ suggested by some algorithm $\varphi.$ In particular, write $N_{i}=N_{i}(T),$ then (11) implies

[TABLE]

We can bound the expected regret from below as follows

[TABLE]

where we used for the last inequality (12) and for the last equality that $T\,l=\sum_{i=1}^{n}\mathbb{E}_{S}N_{i}=\sum_{i\in S}\mathbb{E}_{S}N_{i}+\sum_{i\notin S}\mathbb{E}_{S}N_{i}.$ Now, using Formulas (5) – (7) in Chen & Wang (2018) and Hölder’s resp. Jensen’s inequality as in Section 3.4 of Chen & Wang (2018) one obtains

[TABLE]

The Kullback-Leibler divergence in the latter display can be dealt with by the following lemma which is proved below.

Lemma A.3**.**

For each $S^{\prime}\in\mathbb{A}_{l-1}$ and $i\in S^{\prime}$ the following bound is true

[TABLE]

With Lemma A.3 we have that for any $S^{\prime}\in\mathbb{A}_{l-1}$

[TABLE]

since $\sum_{i\in S^{\prime}}\mathbb{E}_{S^{\prime}}N_{i}\leq Tl.$ Thus, choosing $\varepsilon=\min(C\sqrt{\nicefrac{{n}}{{T}}},1/2)$ for some appropriate small constant $C>0,$ independent of $T,n$ and $l,$ we obtain the assertion. ∎

Proof of Lemma A.3.

Let $\tilde{S}\in\mathbb{A}_{l}$ be arbitrary. Then $\mathbb{P}_{S^{\prime}}(\cdot|\tilde{S})$ denotes the (categorical) probability distribution on the set $\tilde{S}$ parameterized by $\theta_{S^{\prime}},$ i.e.,

[TABLE]

If $i\notin\tilde{S}$ then \mathrm{KL}\big{(}\mathbb{P}_{S^{\prime}}(\cdot|\tilde{S}),\mathbb{P}_{S^{\prime}\cup\{i\}}(\cdot|\tilde{S})\big{)}=0, as both distributions coincide in this case. Thus, we have the following bound

[TABLE]

as $i\in\tilde{S}$ happens $\mathbb{E}_{S^{\prime}}N_{i}$ times in expectation. We proceed by bounding the Kullback-Leibler-divergence on the right-hand side of (13). Define $J_{+}=|\tilde{S}\cap S^{\prime}|,$ and $J_{-}=|\tilde{S}\cap(S^{\prime})^{\complement}|.$ Since $\tilde{S}\in\mathbb{A}_{l}$ it holds that $J_{+}+J_{-}=l.$ With this, the categorical probabilities for $j\in\tilde{S}$ are given by

[TABLE]

For $j\neq i$ it holds that $\nicefrac{{(p_{j}-q_{j})^{2}}}{{q_{j}}}\leq\nicefrac{{8\varepsilon^{2}}}{{l^{3}}}.$ We show this exemplary for the case, where $j\neq i$ and $j\in\tilde{S}\cap S^{\prime},$ while the case $j\neq i$ and $j\notin\tilde{S}\cap S^{\prime},$ can be dealt with similarly. It holds that $J_{+}+(1-\varepsilon)J_{-}=l-\varepsilon J_{-}$ and $J_{+}+1+(1-\varepsilon)(J_{-}-1)=l+\varepsilon(1-J_{-}),$ so that

[TABLE]

and with this

[TABLE]

as the terms inside the squared brackets are respectively greater than $l/2,$ since $\varepsilon\in(0,1/2)$ and $|J_{+}|,|J_{-}|\leq l.$ If $j=i,$ then $\nicefrac{{(p_{j}-q_{j})^{2}}}{{q_{j}}}\leq\nicefrac{{20\varepsilon^{2}}}{{l}}.$ Indeed, we have

[TABLE]

so that

[TABLE]

since \big{(}1-l-\varepsilon(J_{+}-J_{-})\big{)}^{2}\leq 2l^{2}+2\varepsilon^{2}l^{2}\leq\nicefrac{{5l^{2}}}{{2}}. Note that $|p_{j}-q_{j}|<1$ for each case, so that by using Lemma A.1 and $l\geq 2$ we obtain for Equation (13) that

[TABLE]

which completes the proof. ∎

Proof of Theorem 4.2 (i).

Let $\varphi$ be some arbitrary algorithm suggesting the subsets $(S_{t}^{\varphi})_{t\in[T]}\subset\mathbb{A}_{full}.$ In the following we define two problem instances characterized by score parameters $\theta^{(1)},\theta^{(2)}\in\Theta$ such that

[TABLE]

where the infimum is taken over all terminating algorithms $\varphi$ for the flexible Pre-bandit problem and $\check{C}>0$ is a constant similar to $C$ as in the assertion. The proof will be then complete due to

[TABLE]

Thus, we proceed by showing (14).

The observation at $t$ under the PL model assumption for the algorithm $\varphi$ for an instance with score parameter $\theta$ is a random sample of $P_{S_{t}^{\varphi},\theta}=P_{S_{t}^{\varphi}},$ where

[TABLE]

The probability distribution with respect to $\varphi$ and $\theta$ is denoted by $\mathbb{P}_{\theta}^{\varphi}=\mathbb{P}_{\theta}$ and the corresponding expectation by $\mathbb{E}_{\theta}^{\varphi}=\mathbb{E}_{\theta}.$ The regret of $\varphi$ for a PL model with parameter $\theta$ over the time horizon $T$ is

[TABLE]

where $N_{S}(t)=\sum_{s=1}^{t}1_{\{S_{s}^{\varphi}=S\}}$ denotes the number of times the subset $S\in\mathbb{A}_{full}$ was suggested by $\varphi$ till time $t\in[T].$ Note that we suppressed here the dependency of $S^{*}$ on $\theta$ in the notation for sake of brevity.

Next, define

[TABLE]

where $\varepsilon\in(0,1-\theta_{min})$ is a hardness parameter of the instances, which will be specified below. Note that both score parameters are elements of $\Theta$ and only differ in two of the $n$ components. It is easy to see that for any $S\in\mathbb{A}_{full}\backslash\{1\}$ and $S^{\prime}\in\mathbb{A}_{full}\backslash\{2\}$ one has that

[TABLE]

Indeed, recall the decomposition of $\theta$ in (5) and obtain

[TABLE]

The inequality $\mathrm{U}(\{2\},\theta^{(2)})-\mathrm{U}(S^{\prime},\theta^{(2)})\geq\min\{1,1/\gamma\}\,\varepsilon$ can be shown similarly. Clearly, the optimal subset to suggest for the problem instance characterized by $\theta^{(1)}$ is $\{1\},$ while $\{2\}$ is optimal for the other scenario associated with $\theta^{(2)}.$ Suggesting other subsets respectively results in an at least linear regret in the hardness parameter $\varepsilon.$ By means of representation (16) and (18) it follows that for $i=1,2$

[TABLE]

The inequalities are intuitive: if the optimal set $\{1\}$ for the parameter $\theta^{(1)}$ is suggested at most $T/2$ times, then one obtains a regret of at least $\varepsilon$ for the suggested sets in the remaining cases, which occur at least $T/2$ times. Similarly, if the suboptimal set $\{1\}$ for the problem instance with $\theta^{(2)}$ is suggested at least $T/2$ times, then one obtains a regret of at least $\varepsilon$ in all these timesteps. The latter display implies

[TABLE]

where we used in the last line a version of Pinkser’s inequality, see Theorem 14.2 in Lattimore & Szepesvári (2020).

We proceed by analyzing the Kullback-Leibler distance in the latter display by means of Lemma A.1 and the following decomposition of the Kullback-Leibler divergence for the family of probability distributions $(\mathbb{P}_{\theta}^{\varphi})_{\theta\in\Theta}$ which can be shown analogously to Lemma 15.1 in Lattimore & Szepesvári (2020).

Lemma A.4**.**

Let $\theta,\theta^{\prime}\in\Theta,$ then

[TABLE]

Note that by definition of the score parameters in (17) it holds that \mathrm{KL}\big{(}P_{S,\theta^{(1)}},P_{S,\theta^{(2)}}\big{)}=0 for any subset $S\in\mathbb{A}_{full}$ which does not contain $\{1\}$ and $\{2\},$ as both distributions are the same for such subsets. For the remaining subsets $S^{\prime}$, which are of order $\mathcal{O}(2^{n-2})$ many, Lemma A.1 yields \mathrm{KL}\big{(}P_{S^{\prime},\theta^{(1)}},P_{S^{\prime},\theta^{(2)}}\big{)}\leq{2\theta_{min}^{-1}\varepsilon^{2}} (cf. the proof of Lemma A.3). We distinguish two cases in the following.

Case 1: $T>2^{n}-1.$

As $\sum_{S\in\mathbb{A}_{full}}\mathbb{E}_{\theta}(N_{S}(T))=T$ for any $\theta\in\Theta$ it is true that $\mathbb{E}_{\theta}(N_{S}(T))\leq\nicefrac{{T}}{{2^{n}-1}}$ for each $S\in\mathbb{A}_{full}$ by the pigeonhole principle. Thus, by means of Lemma A.4 obtain \mathrm{KL}\big{(}\mathbb{P}_{\theta^{(1)}},\mathbb{P}_{\theta^{(2)}}\big{)}\leq\widetilde{C}\,T\varepsilon^{2}, where $\widetilde{C}>0$ is some constant independent of $n$ and $T.$ Hence,

[TABLE]

Case 2: $T\leq 2^{n}-1.$

In this case, note that there are at least $2^{n}-1-T$ many zero summands in $\sum_{S\in\mathbb{A}_{full}}\mathbb{E}_{\theta}(N_{S}(T))$ as the sum equals $T.$ Therefore, similar to the case before obtain by means of Lemma A.4 that \mathrm{KL}\big{(}\mathbb{P}_{\theta^{(1)}},\mathbb{P}_{\theta^{(2)}}\big{)}\leq\widetilde{C}T\varepsilon^{2} for some constant $\widetilde{C}>0$ independent of $n$ and $T.$ Consequently,

[TABLE]

By choosing in both cases $\varepsilon=\min(\bar{C}\,\sqrt{1/T},\,1-\theta_{min})$ for some appropriate constant $\bar{C}>0$ we obtain the assertion with some constants $C,C^{\prime}>0$ which are independent of $T,l$ and $n.$ ∎

Proof of Theorem 4.2 (ii).

For the gap-dependent lower bound we will make use of the following result, which is Lemma 1 in Kaufmann et al. (2016).

Lemma A.5**.**

Let $\nu$ and $\nu^{\prime}$ be two MAB models with $n$ arms and $\nu_{i}$ resp. $\nu_{i}^{\prime}$ denotes the reward distribution for arm $i\in[n]$ respectively. Let $A_{t}$ denote the arm played at round $t$ and $R_{t}$ be the corresponding observed reward. Moreover, let $\mathcal{F}_{t}=\sigma(A_{1},R_{1},\ldots,A_{t},R_{t})$ be the sigma algebra generated by the observations till time instance $t.$ Suppose that $\nu_{i}$ and $\nu_{i}^{\prime}$ are mutually absolutely continuous for each $i\in[n],$ then it holds that

[TABLE]

for any $\mathcal{F}_{T}$-measurable random variable $\mathcal{E}.$ Here, $d(x,y)=x\log(\nicefrac{{x}}{{y}})+(1-x)\log(\nicefrac{{(1-x)}}{{(1-y)}})$ and $N_{i}(t)=\sum_{s=1}^{t}1_{i_{s}^{\varphi}=i}$ is the number of times an algorithm $\varphi$ plays arm $i$ till time instance $t.$

In the following, we will adapt the proof of Theorem 3 in (Saha & Gopalan, 2019b) to our case, which boils down to incorporating our (different) notion of regret into their proof.

To make use of Lemma A.5 we embed the flexible Pre-Bandit problem into a classical MAB problem by considering each subset $S\in\mathbb{A}_{full}$ as an arm. Moreover, we define the score parameters

[TABLE]

where $\Delta\in(0,1-\theta_{min})$ and $\varepsilon>0$ and the $i$-th component of $\theta^{(i)}$ is $1+\varepsilon.$ For $\theta\in\Theta$ and $S\in\mathbb{A}_{full}$ let $P_{S,\theta}$ denote the categorical distribution as in (15). Using Lemma A.5 with $\nu_{S}=P_{S,\theta^{(1)}}$ and $\nu_{S}^{\prime}=P_{S,\theta^{(i)}}$ for $i\neq 1$ for any $S\in\mathbb{A}_{full}$ as the reward distributions of the arms and the $\mathcal{F}_{T}$-measurable random variable $\mathcal{E}=\nicefrac{{N_{\{i\}}(T)}}{{T}},$ one has that

[TABLE]

Now, since $d(x,y)\geq(1-x)\log(\nicefrac{{1}}{{(1-y)}})-\log(2)$ derive that

[TABLE]

As we assume that $\varphi$ is a no-regret algorithm, we have that $\mathbb{E}_{\theta^{(1)}}[N_{\{i\}}]=o(T^{\alpha})$ and $T-\mathbb{E}_{\theta^{(i)}}[N_{\{i\}}]=\mathbb{E}_{\theta^{(i)}}[\sum_{S\in\mathbb{A}_{full},S\neq\{i\}}N_{\{i\}}]=o(T^{\alpha})$ for some $\alpha\in(0,1].$ Hence, by dividing the latter display by $\log(T)$ and by considering $T\to\infty$ one obtains

[TABLE]

Hence, dividing (20) by $\log(T)$ and considering the limit case obtain

[TABLE]

The Kullback-Leibler divergence in (21) can be bounded by the following lemma, which first statement can be shown by following the lines of display (2) in Saha & Gopalan (2019b), while the second statement is straightforward from the choice of the score parameters in (19).

Lemma A.6**.**

For each $i\neq 1$ it holds that

[TABLE]

Moreover, if $i\notin S$ or if $|S|=1,$ then

[TABLE]

Using Lemma A.6 we can derive from (21) by multiplying with $\nicefrac{{(1-\Delta)^{2}}}{{(\Delta+\varepsilon)}}$ that

[TABLE]

Summing over $i\in\{2,\ldots,n\}$ and taking the limit $\varepsilon\to 0$ in the latter display leads to

[TABLE]

Next, we bound the cumulative regret in (9) for any algorithm $\varphi$ for the flexible Pre-Bandit problem from below. For this purpose recall the decomposition in (5) and denote the $i$th component of $\theta^{(1)}$ by $\theta^{(1)}_{i}$ and let $v_{i}^{(1)}=(\theta^{(1)}_{i})^{1/\gamma}.$ Hence, we get

[TABLE]

where we used Lemma A.2 for the inequality together with \sum_{i=1}^{n}1_{i\in S}\big{(}v_{i}^{(1)}\big{)}^{\gamma}\leq|S|. With this obtain from (22) that if $\varphi$ is a no-regret algorithm, then

[TABLE]

which concludes the proof as $\Delta$ corresponds to $\min_{i\notin S^{*}}\theta_{max}-\theta_{i}$ for $\theta=\theta^{(1)}$ and $(1-\alpha)(1-\Delta)^{2}$ is some constant independent of $T$ and $n.$ ∎

Appendix B Proof of Theorem 5.1

We start by introducing the notation for the rest of the proof and recalling the main terms of the TRCB algorithm. Thereafter we give an outline of the proof, before deriving the details.

B.1 Notation and relevant terms

Throughout $(S_{t})_{t=1,\ldots,T}$ denotes the suggested subsets (the preselections) of the TRCB algorithm at each time instance respectively and $(i_{t})_{t=1,\ldots,T}$ the corresponding decisions of the selector, i.e., $i_{t}\in S_{t}.$ Furthermore, let $\gamma\in(0,\infty)$ be some arbitrary degree of preciseness. Next, we clarify the notation as well as recall the main terms emerging in the TRCB algorithm. We define

[TABLE]

to denote the number of times $i$ has been picked by the selector till time instance $t,$ when $i$ and $j$ were both part of the preselection, while $\overline{w}_{i,j}(t):=w_{i,j}(t)+w_{j,i}(t)$ is the number of times either $i$ or $j$ was picked till time instance $t,$ when both were part of the preselection. The relative scores in (4) are estimated in time instance $t$ by

[TABLE]

The arm with the most picks till time instance $t$ is

[TABLE]

Note that in the following we will suppress its dependency on the time instance $t$ in the notation. The (thresholded) random value inside the confidence region of $\hat{O}_{i,J}(t)$ is

[TABLE]

for $i\neq J$ and $\hat{O}^{TRCB}_{i,J}(t)=1$ for $i=J,$ where

[TABLE]

and $C_{shrink}\in(0,1/2)$ is some finite constant. Note that the $\beta_{i}$’s are mutually independent. Recall the definition of regret for any time instance $t\in[T]$ in (9). Due to (10) we will consider the following scaled regret per time

[TABLE]

Finally, let $\mathcal{F}_{t}$ denote the $\sigma$-algebra generated by $S_{1},i_{1},\ldots,S_{t-1},i_{t-1}$ in time instance $t,$ with $\mathcal{F}_{1}$ being the trivial $\sigma$-algebra. Note that $J(t)$ as well as $\overline{w}_{i,J}(t)$ resp. $c_{i,J}$ are $\mathcal{F}_{t}$-measurable for any $t\in[T].$

B.2 Outline of the proof

We introduce in the following the core lemmas to prove the main result, which will be gradually verified in the next subsection. For $t\in[T]$ define

[TABLE]

Thus, $A_{t}$ is the event on which the estimates for the relative scores for arms in the chosen preselection and the optimal preselection with respect to the currently most winning arm $J$ are not close enough to their actual relative score, where the length of the confidence region $c_{i,J}(t)$ determines how closeness is to be understood in this case.

As a consequence, one wishes that the probability that $A_{t}$ happens is sufficiently small. The following lemma establishes this requirement.

Lemma B.1**.**

It holds that

[TABLE]

where the constant in the $\mathcal{O}$-term is independent of $T,l$ and $n.$ In particular, for any $i\in S_{t}\cup S^{*},$

[TABLE]

Next, we investigate the deviation between the scaled regret per time (cf. (26)) and its empirical counterpart. For this purpose, note that

[TABLE]

since $\widetilde{\mathrm{U}}(S^{*};\hat{O}_{J}^{\mathrm{TRCB}})-\widetilde{\mathrm{U}}(S_{t};\hat{O}_{J}^{\mathrm{TRCB}})\leq 0,$ by the definition of $S_{t}$ in line 11 of the TRCB algorithm. Here, we abbreviated $\hat{O}_{J}^{\mathrm{TRCB}}=(\hat{O}_{1,J}^{\mathrm{TRCB}},\ldots,\hat{O}_{n,J}^{\mathrm{TRCB}}).$

The following lemma gives a bound on the ratio between the two terms in squared brackets on the right-hand side of the latter display.

Lemma B.2**.**

Conditioned on $\mathcal{F}_{t}$ there exist constants $C_{1},C_{2}>0$ depending if at all on $\theta_{min}$ and $\gamma$ (but independent of $T,l$ and $n$) such that on $A_{t}^{\complement}$ it holds with probability at least $1-\frac{C_{1}}{\sqrt{t}}$ that

[TABLE]

Moreover, $C_{2}$ is of the form $\mathrm{const}\cdot\theta_{min}^{-2(3+\gamma)}.$ In particular, with probability at least $1-\frac{C_{1}}{\sqrt{t}}$

[TABLE]

The next pillar of the proof is to transfer the high concentration of $\hat{O}_{J}^{\mathrm{TRCB}}$ around $O_{J}$ to a high concentration of the corresponding utilities $\widetilde{\mathrm{U}}$ by exploiting its Lipschitz smoothness.

Lemma B.3**.**

For any $t\in[T]$

[TABLE]

Finally, an upper bound on the expected length of the confidence regions over time (that is basically $(\overline{w}_{i,J}(t))^{-1/2}$) has to be verified.

Lemma B.4**.**

The following statement is valid,

[TABLE]

Conclusion: Proof of Theorem 5.1

Given these core lemmas, we are now in the position to verify Theorem 5.1.

Let $\theta\in\Theta$ and $T\in\mathbb{N}$ with $T>n,$ then since $r(S_{t})\leq\tilde{r}(t),$ for any $t\in[T],$ we have

[TABLE]

where we used the tower property of the conditional expected value. Note that $\tilde{r}\leq 1/\theta_{min}$ such that by applying Lemma B.2, Lemma B.1 and then Lemma B.3, one can derive that

[TABLE]

where $C_{i}>0,$ for $i\in\{0,1,2,3\},$ are constants depending if at all only on $\theta_{min}$ and $\gamma,$ but are independent of $T,l$ and $n$. Next, since $\sum_{t=1}^{T}t^{-1/2}\leq 2\sqrt{T}$ and $\log(l\cdot t)\leq 2\log(T),$ due to $l\leq n<T,$ we can further estimate the right-hand side of the latter display to obtain

[TABLE]

where we used Lemma B.4 for the second last inequality. Here, the constants $C_{4},C_{5},C_{6}>0$ are as before depending (if at all) on $\theta_{min}$ and $\gamma,$ but are independent of $T,l$ and $n$. In particular, we have $C_{4}$ is of the form $\mathrm{const}\cdot\theta_{min}^{-1},$ while $C_{6}$ is of the form

[TABLE]

This concludes the proof.

B.3 Proofs of the core lemmas in Subsection B.2

We start with the proof of Lemma B.1. For this we need the following result, which is Lemma 1 in Saha & Gopalan (2019a).

Lemma B.5**.**

It holds that for any $r\in\mathbb{N},i,j\in[n]$ and $\varepsilon>0$ that

[TABLE]

Proof of Lemma B.1.

Define the function $\phi(x)=x^{-1}-1,$ then note that \phi\big{(}\frac{w_{j,i}(t)}{\overline{w}_{i,j}(t)}\big{)}=\hat{O}_{i,j}(t) and \phi\big{(}\frac{\theta_{j}}{\theta_{i}+\theta_{j}}\big{)}=O_{i,j}. Further, by the mean value theorem there exists for any pair of arms $(i,j)$ some $\tilde{z}_{i,j}$ between $\frac{w_{j,i}(t)}{\overline{w}_{i,j}(t)}$ and $\frac{\theta_{j}}{\theta_{i}+\theta_{j}}$ such that

[TABLE]

Note that

[TABLE]

and in particular if $j=J$ then

[TABLE]

as $\overline{w}_{i,J}\leq 2w_{J,i}$ by definition of $J$ and $\theta_{min}<1.$ Let us write E_{i,J}(t)=\big{|}\hat{O}_{i,J}(t)-O_{i,J}\big{|} for sake of brevity, then we get with the deviation above for $\varepsilon>0$ for any $t\in[2,T]\cap\mathbb{N}$ that

[TABLE]

where Lemma B.5 was used in the last step. Setting $\varepsilon=\sqrt{\nicefrac{{8\,\log(l\,t^{3/2})}}{{\theta_{min}^{4}}}}$ in the last display, we obtain in combination with the law of total expectation that conditioned on $\mathcal{F}_{t}$ that

[TABLE]

where we used that thresholding of the relative scores only makes the probability of the event smaller for the first inequality and in the second last step that, firstly, $C_{shrink}\leq 1/2$ in combination with $1/2\,c_{i,J}(t)=\nicefrac{{\varepsilon}}{{\sqrt{\overline{w}_{i,J}(t)}}}$ and secondly, that $|S_{t}\cup S^{*}|\leq 2l.$ The constant in the $\mathcal{O}$-term is independent of $l,$ $T$ and $n.$ This concludes the lemma. ∎

Proof of Lemma B.2.

Let us write $S^{*}_{2}(O)=\sum_{i\in S^{*}}O_{i,J}^{\frac{1+\gamma}{\gamma}}$ and $S^{*}_{1}(O)=\sum_{i\in S^{*}}O_{i,J}.$ In the same spirit define $S^{*}_{2}(\hat{O}),$ $S^{*}_{1}(\hat{O})$, $S^{t}_{2}(O),$ $S^{t}_{1}(O),$ $S^{t}_{2}(\hat{O})$ and $S^{t}_{1}(\hat{O}),$ where $\hat{O}$ is short for $\hat{O}_{J}^{\mathrm{TRCB}}.$ Then,

[TABLE]

It holds that

[TABLE]

Hence, all of the latter terms can be bounded from below resp. above by $\tilde{C}_{j}/l$ for some suitable constants $C_{j}$ which depend if at all on $\theta_{min}.$ Following the lines of proof of Lemma B.1, it can be shown that there exists a constant $C_{1}>0$ (depending on $\theta_{min}$ and $\gamma$) such that the ratios of the terms in the squared brackets in (29) are bounded by some constant $C_{2}>0$ on the event $A_{t}^{\complement},$ with probability at least $1-\frac{C_{1}}{\sqrt{t}}.$ Hence, the whole term in (29) can be bounded with probability at least $1-\frac{C_{1}}{\sqrt{t}}$ by some constant $C_{3}>0$ which if at all depends only on $\theta_{min}.$ This yields the first part of the lemma. The second part is just a consequence of the first part together with (28). ∎

Proof of Lemma B.3.

Define the function $\phi(x_{1},\ldots,x_{l})=\nicefrac{{\sum_{i=1}^{l}x_{i}^{\nicefrac{{(1+\gamma)}}{{\gamma}}}}}{{\sum_{i=1}^{l}x_{i}}}$ for $x_{1},\ldots,x_{l}\in[A,B]$ for $0<A<B.$ Then, we have that for $i=1,\ldots,l$

[TABLE]

It can be easily checked that

[TABLE]

Without loss of generality assume that $S_{t}=\{1,\ldots,l\},$ then by setting $x_{i}=O_{i,J}$ and $y_{i}=O_{i,J}^{\mathrm{TRCB}}(t)$ and noting that $\phi(x_{1},\ldots,x_{l})=\widetilde{\mathrm{U}}(S_{t};O_{J})$ as well as $\phi(y_{1},\ldots,y_{l})=\widetilde{\mathrm{U}}(S_{t};\hat{O}_{J}^{\mathrm{TRCB}}),$ we obtain with the mean value theorem that

[TABLE]

by choosing $C=\max\{\nicefrac{{\theta_{min}^{\frac{\gamma-1}{\gamma}}}}{{\gamma}},\nicefrac{{\theta_{min}^{\frac{1-\gamma}{\gamma}}}}{{\gamma}},\},$ since $\theta_{min}\leq O_{i,J}\leq 1/\theta_{min}$ and $\theta_{min}\leq O_{i,J}^{\mathrm{TRCB}}(t)\leq 1/\theta_{min}.$ ∎

Proof of Lemma B.4.

Since $\sum_{t=1}^{T}t^{-1/2}\leq 2\sqrt{T}$ one has $\sum_{\overline{w}_{i,J}(t)=1}^{\overline{w}_{i,J}(T)}\nicefrac{{1}}{{\sqrt{\overline{w}_{i,J}(t)}}}\leq 2\sqrt{\overline{w}_{i,J}(T)}.$ Due to $\sum_{i\in[n]}\mathbb{E}\overline{w}_{i,J}(T)\leq T$ it follows by Jensen’s inequality that $\sum_{t=1}^{T}\mathbb{E}\,\sum_{i\in S_{t}}\sqrt{\frac{1}{\overline{w}_{i,J}(t)}}\leq 4\sqrt{T\,n}.$ ∎

Appendix C Proof of Theorem 5.2

We start by introducing the notation for the rest of the proof and recalling the main terms of the CBR algorithm. Thereafter we give an outline of the proof, before deriving the technical details.

We break the proof down into two core lemmas, for which we first clarify the notation. We assume that without loss of generality $|S^{*}|=1,$ i.e., there is only one best arm, as this makes the learning problem only more difficult. Indeed, having several arms with the same highest score extends the opportunities to identify one of these highest score arms. To ease the notation we denote the score of the highest scored arm with $\theta_{max},$ which is $1$ by definition of $\Theta$ and its index by $i_{max}.$

C.1 Notation and relevant terms

We define the estimate for the pairwise winning probability $q_{i,j}$ (cf. (3)) by

[TABLE]

where $w_{i,j}$ are as in (23) and with the convention that $\nicefrac{{x}}{{0}}=0.$ With $J(t)=J$ we again denote the arm (within the active set) with the most picks till time instance $t$ as in (25). With $\Delta_{i}=\theta_{max}-\theta_{i}$ we define the gap between the score of the $i$th arm and the overall best arm. The lengths of the confidence intervals are

[TABLE]

thereby implicitly setting $\overline{w}_{ii}(t)=\infty$ for any $i\in[n].$

C.2 Outline of the proof

We define the following events

[TABLE]

Here, $B_{t}$ is the event where an arm exists whose pairwise probability estimate for winning against $J$ is not close enough to its actual parameter, where closeness is understood by means of the confidence length $c_{i,J}(t).$ $R_{t}$ is the event when the most winning arm $J$ is not the overall best arm and $E_{t}$ is the event, where the offered subset at time instance $t$ is not a singleton. All these events are ”bad” events and we will show that their probability of occurrence is sufficiently small.

We have the following key lemmas to prove the main result.

Lemma C.1**.**

There exist constants $C_{1},C_{2},C_{3}>0$ independent of  $T$ and $n$ (depending if at all on the parameter space $\Theta$), such that

[TABLE]

Lemma C.2**.**

There exist constants $C_{1},C_{2}>0$ independent of  $T$ and $n$ (depending if at all on the parameter space $\Theta$), such that

[TABLE]

Lemma C.3**.**

*It holds that *

[TABLE]

Putting all together.

Recalling the cumulative regret in (9), we obtain

[TABLE]

where we used Lemma C.1 and Lemma C.2 to derive the constants $C_{0},C_{1}>0,$ which are both independent of $T$ and $n,$ while Lemma C.3 introduced the factor accompanying $C_{1}.$ Furthermore, we used that on $R_{t}^{\complement}\cap E_{t}^{\complement}$ we have that $S_{t}$ equals $\{i_{max}\}=S^{*}$ and thus $r(S_{t})=0.$

C.3 Proofs of the core lemmas in Subsection C.2

Proof of Lemma C.1.

Using Lemma B.5 one obtains

[TABLE]

By summing over $t$ till $T,$ we get $\sum_{t=1}^{T}\nicefrac{{2}}{{t^{5}}}<2\sum_{t=1}^{\infty}\nicefrac{{1}}{{t^{2}}}=\pi^{2}/3,$ which yields the first claim.

For the second claim, let $A_{t}$ denote the set of active arms at time instance $t,$ i.e.,

[TABLE]

It holds that conditioned on $B_{t}^{\complement}$ we have that $i_{max}\in A_{t}$ almost surely. Indeed,

[TABLE]

where we used that $\sigma(x)\leq 0$ iff $x\leq 0$ and for the last inequality that $\hat{q}_{i_{max},J}(t)+c_{i_{max},J}(t)\geq q_{i_{max},J}(t)$ on $B_{t}^{\complement},$ while $q_{i_{max},J}(t)>1/2$ holds by definition of $i_{max}.$

Next, consider the counting process $M_{t}^{i,i_{max}}:=w_{i,i_{max}}-w_{i_{max},i}$ for some $i\in A_{t}\backslash\{i_{max}\}$ and define for sake of brevity the event $\tilde{S}_{s}^{i}=\{\{i,i_{max}\}\in S_{s}\}.$ Note that $M_{t}^{i,i_{max}}$ can be written as

[TABLE]

It holds that the event $\{\{i,i_{max}\}\in S_{s}\}$ has a strictly positive probability for any arm $i\in A_{t}\backslash\{i_{max}\}$ and any $s\in[t],$ as otherwise the arm would not be active anymore. Conditioned on some set $S_{s}$ we have that

[TABLE]

where $H^{\prime}=\sum_{i\in[n]}\theta_{i}.$ Thus, we can find a constant $C>0,$ which depends only on $\Theta$ such that for each $s\in[t]$

[TABLE]

Therefore, $\mathbb{E}M_{t}^{i,i_{max}}\leq-(t-1)\,C\,\Delta_{i}$ and by Lemma C.5 it follows that

[TABLE]

The event $R_{t}$ is contained in the event that there exists an active arm $i$ such that the winning count of $i_{max}$ against $i$ is smaller than the winning count of $i$ against $i_{max},$ that is $M_{t}^{i,i_{max}}\geq 0.$ Hence, using the union bound in combination with the latter display we obtain

[TABLE]

from which we can conclude the lemma. ∎

Proof of Lemma C.2.

For any $i\neq i_{max}$ we have that

[TABLE]

Now, similar as in the proof of Lemma C.1 before, we can find a constant $\tilde{C}>0$ which depends if at all on $\Theta$ such that $\mathbb{P}(i_{s}\in\{i,i_{max}\},\{i,i_{max}\}\in S_{s})\geq\theta_{min}\tilde{C}$ for any active arm $i$ and each $s\in[t].$ With this, we obtain that $\mathbb{E}(\overline{w}_{i,i_{max}}(t))\geq(t-1)\theta_{min}\tilde{C}.$ Using Lemma C.6 with $\overline{w}_{i,i_{max}}$ as the counting process one can derive that there exists a constant $C>0$ depending on $\Theta$ such that

[TABLE]

Next, write for short $\delta_{i,i_{max}}=\hat{q}_{i,i_{max}}(t)+c_{i,i_{max}}(t)-1/2$ and note that

[TABLE]

where we used that $J(t)=i_{max}$ on $R_{t}^{\complement}$ for the first inequality, $\sigma(x)\leq 0$ iff $x\leq 0$ for the second inequality, for the third inequality that $\hat{q}_{i,i_{max}}(t)-c_{i,i_{max}}(t)\leq q_{i,i_{max}}(t)$ on $B_{t}^{\complement},$ while the last inequality is due to $\log(nt^{3/2})\leq 5/2\log(T),$ as $\max\{n,t\}\leq T.$ One can find constants $C_{i}\in[1/4,1/2]$ such that $1/2-q_{i,i_{max}}=C_{i}\Delta_{i}.$ Indeed, note that $1/2-q_{i,i_{max}}=\nicefrac{{\Delta_{i}}}{{(2(\theta_{i}+\theta_{max}))}}$ and it holds that

[TABLE]

Hence, with these considerations one obtains

[TABLE]

Now, the summation over $t$ on the right-hand side of the last display is such that $\nicefrac{{20\log(T)}}{{C_{i}^{2}\Delta_{i}^{2}}}\leq\nicefrac{{(t-1)C}}{{2}}.$ Thus, we can use (30) to further estimate the last display by

[TABLE]

for some constants $C_{1},C_{2}>0.$ From the latter display we can conclude the lemma. ∎

Proof of Lemma C.3.

Note that

[TABLE]

Since $\theta\in\Theta$ it holds that $\sum_{i\in S_{t}}\theta_{i}\geq\theta_{min}.$ With this, and the fact that $\theta_{i}\leq\theta_{max}=1,$ we can infer that

[TABLE]

Considering the function $f(x)=x^{1/\gamma}$ defined for $x\in[\theta_{min},\theta_{max}]$ the assertion follows easily by the mean-value theorem as in the proof of Lemma B.3. ∎

C.4 Technical results

In this subsection we collect the technical auxiliary results needed for the proofs of the core lemmas. These technical results could also be of independent interest.

The next two lemmas were of major importance for the proof of Lemma C.1.

Lemma C.4**.**

Let $M_{t}=\sum_{s=1}^{t}Z_{s},$ where $(Z_{s})_{s=1,\ldots,t}$ are random variables with values in $\{-1,0,1\},$ such that $\mathcal{F}_{s}$ is the canonical filtration generated by $\{Z_{1},\ldots,Z_{s-1}\}$ and $Z_{s+1}$ is conditionally independent of $Z_{s+2},\ldots,Z_{t}$ given $\mathcal{F}_{s}.$ We have that for any $z>0$

[TABLE]

Proof of Lemma C.4.

The function $f(z_{1},\ldots,z_{t})=z_{1}+\ldots+z_{t}$ is Lipschitz-continuous with Lipschitz constant $L=2$ if $-1\leq z_{i}\leq 1$ for each $i.$ It is a well-known result that the sequence of random variables $(X_{i})_{i=1,\ldots,t}$ with $X_{i}=\mathbb{E}[f(Z_{1},\ldots,Z_{t})|\mathcal{F}_{i}]$ is a martingale (the so-called Doob martingale) with bounded differences $|X_{i+1}-X_{i}|\leq 2L=4$ (cf. Lemma 11 in Kocsis et al. (2006)). Consider the martingale difference sequence $\tilde{X}_{i}=X_{i}-\mathbb{E}X_{i}=X_{i}-\mathbb{E}M_{t}$ and note that $\tilde{X}_{t}=X_{t}-\mathbb{E}X_{t}=M_{t}-\mathbb{E}M_{t}$ and $\tilde{X}_{0}=0$ by setting $\mathcal{F}_{0}=\{\varnothing,\Omega\}.$ Thus, the Azuma-Hoeffding inequality implies for any $z>0$ that

[TABLE]

∎

Lemma C.5**.**

Consider the setting of Lemma C.4 and assume that there exists $\Delta_{t}$ such that $\mathbb{E}(M_{t})\leq\Delta_{t}/2.$ Then,

[TABLE]

Proof of Lemma C.5.

[TABLE]

where we used Lemma C.4 in the last step. ∎

For the proof of Lemma C.2 we use the following variant of Lemma 13 in Kocsis et al. (2006).

Lemma C.6**.**

Let $N_{t}=\sum_{s=1}^{t}Z_{s},$ where $(Z_{s})_{s=1,\ldots,t}$ are random variables with values in $\{0,1\},$ such that $\mathcal{F}_{s}$ is the canonical filtration generated by $\{Z_{1},\ldots,Z_{s-1}\}$ and $Z_{s+1}$ is conditionally independent of $Z_{s+2},\ldots,Z_{t}$ given $\mathcal{F}_{s}.$ If $\mathbb{E}N_{t}\geq 2\Delta_{t},$ for some $\Delta_{t}$ then

[TABLE]

Proof of Lemma C.6.

By using $\mathbb{E}N_{t}\geq 2\Delta_{t},$ we have

[TABLE]

where we used Lemma 12 of Kocsis et al. (2006) for the last inequality. ∎

Appendix D Optimal subsets for restricted Pre-Bandits and an efficient algorithm for utility maximization

In this section, we show that the best arm is always element of the optimal preselection for the restricted Pre-Bandit case. Following this, we present a sophisticated algorithm (Algorithm 3) to avoid highly computational costs for determining the maximizing set in line 11 of Algorithm 1.

The following lemma, which can be verified by simple techniques of curve sketching, is the foundation for Algorithm 3 and the proof of Lemma D.2.

Lemma D.1**.**

Let $0\leq a<b$ be real values, $(\theta_{1},\ldots,\theta_{n})\in[a,b]^{n}$ and $S\subseteq[n]$ be a nonempty subset. Further, define $f_{\gamma}:[a,b]\to\mathbb{R}^{+}$ by

[TABLE]

The following statements are valid.

1. (i)

For $\tilde{\theta}=\nicefrac{{\sum_{i\in S}\theta_{i}^{1+\gamma}}}{{\sum_{i\in S}\theta_{i}^{\gamma}}}$ we have that $f_{\gamma}(\tilde{\theta})=f_{\gamma}(0)=\tilde{\theta}.$ 2. (ii)

$f_{\gamma}$* has a unique global minimum in $\bar{\theta},$ which is the (unique) real-valued solution of the following equation in $x$*

[TABLE]

It holds that $f_{\gamma}$ is strictly decreasing in $[a,\bar{\theta}]$ and strictly increasing in $[\bar{\theta},b].$ Moreover, $\bar{\theta}\leq\tilde{\theta}.$

Lemma D.2**.**

Let $\theta\in\Theta$ be such that $|\underset{i\in[n]}{\operatorname{arg}\,\operatorname{max}}\;\theta_{i}|=1$ and let $J=\underset{i\in[n]}{\operatorname{arg}\,\operatorname{max}}\;\theta_{i}.$ Then, for any $l\in\mathbb{N}$, one has $J\in S^{*},$ where each $S^{*}$ is a maximizing subset as in (7) for $\mathbb{A}=\mathbb{A}_{l}$. Furthermore, if $|\underset{i\in[n]}{\operatorname{arg}\,\operatorname{max}}\;\theta_{i}|>1$ then $\mathrm{U}(\{J\})\geq\mathrm{U}(\{J\}\cup\{i\})$ for any $i\in[n]$, with an equality if and only if $\theta_{i}=\theta_{J}.$ The same holds true for $\widetilde{\mathrm{U}}.$

Proof of Lemma D.2.

We prove the first assertion by contradiction. Hence, suppose that $J\notin S^{*}.$ Let $\tilde{J}\in S^{*}$ be such that $\theta_{\tilde{J}}<\theta_{J}$ and define $\tilde{S}=S^{*}\backslash\{\tilde{J}\}\cup\{J\}.$ Thus, by assumption it should hold that

[TABLE]

In terms of Lemma D.1 this means that $f_{\gamma}(\theta_{\tilde{J}},S^{*}\backslash\{\tilde{J}\})>f_{\gamma}(\theta_{J},S^{*}\backslash\{\tilde{J}\}),$ but this is a contradiction due to (i) and (ii) of Lemma D.1, as $\theta_{J}>\tilde{\theta}=\frac{\sum_{i\in S^{*}\backslash\{\tilde{J}\}}\theta_{i}^{1+\gamma}}{\sum_{i\in S^{*}\backslash\{\tilde{J}\}}\theta_{i}^{\gamma}}$ and $\bar{\theta}\in[0,\tilde{\theta}].$ The second claim follows immediately by the strict monotonic behavior of $f_{\gamma}$ and the claims for $\widetilde{\mathrm{U}}$ can be shown similarly. ∎

Let $\theta_{(i)}$ denote the $i$-th order statistic for $(\theta_{1},\ldots,\theta_{n}),$ i.e.,

[TABLE]

then Lemma D.1 implies that f_{\gamma}\big{(}v;\{\theta_{(n)}\}\big{)}\leq f_{\gamma}\big{(}\theta_{(n)};\{\theta_{(n)}\}\big{)} for any $v\in[0,\theta_{(n)}]$ and the smallest decrease of f_{\gamma}\big{(}\cdot;\{\theta_{(n)}\}\big{)} over the discrete set $\{\theta_{(1)},\ldots,\theta_{(n-1)}\}$ is either for $\theta_{(n-1)}$ or for $\theta_{(1)}.$

With this, Algorithm 3 successively builds a set $S$ which will maximize the expected utility in (6) for a given score parameter $\theta=(\theta_{1},\ldots,\theta_{n})$. First, the scores are sorted in order to find the arms with the highest scores, as by Lemma D.2 these are always element of the maximizing subset. If more than $(l-1)$ elements have the same highest score, a randomly chosen $l$-sized set of these is returned, since the expected utility among all possible $l$-sized subsets of these is the same by Lemma D.1 or Lemma D.2.

Otherwise, an active index set $A$ is initialized containing all indices for which it is not decided yet, if they are part of the maximizing set $S$ eventually. As by Lemma D.2 the expected utility decreases from that point on by enlarging the set $S,$ the algorithm determines the arm with the smallest decrease for the expected utility, where ties are broken arbitrary by two possible candidates.

Since the expected utility of the currently set $S$ is identical to $f_{\gamma}(0;S)$ only the arms with the smallest resp. highest score parameter in $A$ have to be checked by the implication after Lemma D.2. It can be shown that the algorithm has worst complexity of $O(l\,n\,\log(n))$ if an efficient sorting algorithm is used in the initial step.

Appendix E Further experiments for the Pre-Bandit problem

In this section, we provide further experiments on synthetic data for the two variants of the Pre-Bandit problem.

Restricted Pre-Bandit problem (larger number of arms)

First, we present two additional scenarios of the simulation study in Section 6 for the restricted Pre-Bandit problem with larger numbers of arms $n$ and different preselection sizes $l.$ In particular, we investigate the performance of the following algorithms, which were also analyzed in Section 6, for the restricted Pre-Bandit problem:

• •

TRCB: The TRCB algorithm in Algorithm 1 with $C_{shrink}=7\cdot 10^{-5}$ and $\theta_{min}=0.02$ (here as a parameter of the algorithm).

• •

UCB-Oracle: UCB-type algorithm of Agrawal et al. (2016) with knowledge of the best arm in advance and revenues are set to be the estimated score parameters (in short $r=\hat{\theta}$).

• •

UCB-Sampling: UCB-type algorithm of Agrawal et al. (2016) without knowledge of the best arm in advance (sampled with MNL probability among the three best) and $r=\hat{\theta}$.

• •

TS-Oracle: The Thompson sampling algorithm of Agrawal et al. (2017) (Algorithm 1) with knowledge of the best arm in advance and $r=\hat{\theta}$.

• •

TS-Sampling: The Thompson sampling algorithm of Agrawal et al. (2017) (Algorithm 1) without knowledge of the best arm in advance (sampled with MNL probability among the three best) and $r=\hat{\theta}$.

• •

TS-Oracle-Corr: Correlated Thompson sampling algorithm of Agrawal et al. (2017) (Algorithm 2) with knowledge of the best arm in advance and $r=\hat{\theta}$.

The left picture in Figure 2 provides the findings for the case $n=20$ and $l=4,$ while the right picture illustrates our results for $n=30$ and $l=5.$ Both scenarios are considered for the time horizons $T\in\{i\cdot 2000\}_{i=1}^{5}$ and the score parameters are drawn randomly from the $n$-simplex without any restrictions on $\theta_{min}$ and with $\gamma=1.$

The findings are similarly as for the case $n=10$ and $l=3,$ that is only the Thompson Sampling algorithm with knowledge of the best arm apriori (TS-Oracle) outperforms TRCB, while the other algorithms are outperformed by TRCB. Furthermore, we report the empirical standard deviations of the considered algorithms for each time horizon in both scenarios in Table 3. Only TS-Oracle has a throughout smaller standard deviation than TRCB, while all the others have variations of a higher magnitude than TRCB.

Restricted Pre-Bandit problem (Varying degree of preciseness)

Next, we consider two additional scenarios, in which we initially set $\gamma=1/20$ such that the most preferred subsets consists throughout of the top-$l$ arms and The results for $\gamma=1/20$ are depicted in Figure 3 for the cases $(n,l)=(10,3)$ and $(n,l)=(20,4)$ for the algorithms described above. Note that TS-Oracle-Corr could not be compared as it sampled negative values for the score parameters, which lead to numerical issues regarding the evaluation of the utility function. Again the findings are in line with the observations we have made in the simulations before, i.e., only TS-Oracle is able to outperform our algorithm TRCB due to its advantage of knowing the best arm. In particular, this demonstrates that our algorithm performs well for scenarios where top-$l$ subsets are the desired outcome for a user.

In addition, we consider the case $\gamma=20$ such that the most preferred subsets are basically all subsets which contain the arm with the highest score (cf. Example 2 in the main paper). Figure 4 illustrates the results for the cases $(n,l)=(10,3)$ and $(n,l)=(20,4),$ where we do not included the algorithms which have prior knowledge of the best arm, as these naturally have throughout a regret of zero. This experiment indicates that the considered DAS algorithms depend too much on the assumption that the no-choice option corresponds to the highest scored arm as also remarked in Section 6.

Flexible Pre-Bandit problem

In addition to the simulations in Section 6, we investigate the empirical regret growth over time for larger numbers of arms $n$ for our CBR algorithm for the flexible Pre-Bandit problem. We consider two variants of the CBR-algorithm:

• •

CBR: The CBR algorithm with $\sigma(x)=(1\wedge x)1_{[0,\infty)}(x).$

• •

CBR-As: The CBR algorithm with \sigma(x)=\frac{1}{\pi}\arctan\Big{(}\frac{x-\nicefrac{{1}}{{2}}}{(1-x)^{\rho}x^{\rho}}\Big{)}+\frac{1}{2} and $\rho=2.$

Figure 5 illustrates the results of our simulations for both CBR algorithm variants over 500 repetitions, respectively, with $n\in\{60,120,240\},$ over the time horizons $T\in\{i\cdot 2000\}_{i=1}^{5}$ and the score parameters were drawn randomly from the unit interval and with $\gamma=1.$

It is clearly visible that CBR-As outperforms CBR due to the more sophisticated choice of the S-curved function $\sigma.$ Thus, it is reasonable to believe that the performance of CBR can be significantly improved by an appropriate choice of $\sigma.$ Note that the Double Thompson Sampling considered in Section 6 was not competitive in these scenarios and is therefore omitted.