Lexicographic Multiarmed Bandit

TL;DR

This paper introduces algorithms for multiobjective multiarmed bandit problems with lexicographic objectives, achieving bounded and sublinear regret under various prior knowledge settings, and validates them through experiments.

Contribution

It develops novel algorithms for lexicographic multiobjective bandits that achieve bounded and sublinear regret, extending bandit theory to complex multiobjective scenarios.

Findings

Algorithms achieve expected regret uniformly bounded in time.

Proposed methods attain sublinear gap-free regret in prior-free settings.

Experimental results demonstrate effective performance across diverse problems.

Abstract

We consider a multiobjective multiarmed bandit problem with lexicographically ordered objectives. In this problem, the goal of the learner is to select arms that are lexicographic optimal as much as possible without knowing the arm reward distributions beforehand. We capture this goal by defining a multidimensional form of regret that measures the loss of the learner due to not selecting lexicographic optimal arms, and then, consider two settings where the learner has prior information on the expected arm rewards. In the first setting, the learner only knows for each objective the lexicographic optimal expected reward. In the second setting, it only knows for each objective near-lexicographic optimal expected rewards. For both settings we prove that the learner achieves expected regret uniformly bounded in time. The algorithm we propose for the second setting also attains bounded regret for the multiarmed bandit with satisficing objectives. In addition, we also consider the harder prior-free case, and show that the learner can still achieve sublinear in time gap-free regret. Finally, we experimentally evaluate performance of the proposed algorithms in a variety of multiobjective learning problems.

Tables

Table 1. Table 1 : Expected reward vectors.

Setting	${𝝁}_1$	${𝝁}_2$	${𝝁}_3$
Setting 1	$\left(\begin{array}{c}0.50\\ 0.50\end{array}\right)$	$\left(\begin{array}{c}0.50\\ 0.40\end{array}\right)$	$\left(\begin{array}{c}0.40\\ 0.90\end{array}\right)$
Setting 2	$\left(\begin{array}{c}0.50\\ 0.50\end{array}\right)$	$\left(\begin{array}{c}0.50\\ 0.40\end{array}\right)$	$\left(\begin{array}{c}0.40\\ 0.50\end{array}\right)$
Setting 3	$\left(\begin{array}{c}0.50\\ 0.50\end{array}\right)$	$\left(\begin{array}{c}0.50\\ 0.40\end{array}\right)$	$\left(\begin{array}{c}0.40\\ 0.10\end{array}\right)$

Table 2. Table 3 : Regrets of OM-LEX 1, NOM-LEX 1, 2, and 3 in Settings 1, 2, and 3 along with their single objective variants (labeled as SO). 8 8 8 p denotes × 10 − 12 absent superscript 10 12 \times 10^{-12} and f denotes × 10 − 15 absent superscript 10 15 \times 10^{-15} . Average regrets are rounded to three most significant digits and standard deviations are rounded to two most significant digits.

	Setting 1						Setting 2						Setting 3
Learners	Obj. 1			Obj. 2			Obj. 1			Obj. 2			Obj. 1			Obj. 2
OM-LEX 1	12.0	$\pm$	2.1	333	$\pm$	56	321	$\pm$	71	314	$\pm$	61	11.0	$\pm$	2.0	323	$\pm$	60
OM-LEX 1 (so)	334	$\pm$	73				334	$\pm$	73				334	$\pm$	73
NOM-LEX 1	1210	$\pm$	700	1150	$\pm$	680	4450	$\pm$	3800	2400	$\pm$	2900	285	$\pm$	110	270	$\pm$	120
NOM-LEX 2	1250	$\pm$	630	1320	$\pm$	600	1240	$\pm$	600	1160	$\pm$	660	14.9	$\pm$	12	4990	$\pm$	3000
NOM-LEX 3	12.7	$\pm$	7.0	1250	$\pm$	640	253	$\pm$	140	269	$\pm$	140	8.38	$\pm$	5.6	245	$\pm$	140
NOM-LEX 1 (so)	706	$\pm$	770				706	$\pm$	770				706	$\pm$	770
PF-LEX 1	764	$\pm$	210	723	$\pm$	1.1p	806	$\pm$	240	723	$\pm$	1.1p	679	$\pm$	77	723	$\pm$	1.1p
PF-LEX 2	9820	$\pm$	4.5	52.8	$\pm$	14f	5000	$\pm$	860	94.6	$\pm$	24	52.8	$\pm$	14f	105	$\pm$	32

Table 3. Table 4 : Prior knowledge and parameters of the algorithms for additional experiments.

Algorithm	Prior Knowledge & Parameters
OM-LEX 2	${\mu }_{\ast }^1={\mu }_{\ast }^2={\mu }_{\ast }^3=0.50$
NOM-LEX 4	${\eta }_1={\eta }_2={\eta }_3=0.45$
NOM-LEX 5	${\eta }_1={\eta }_3=0.45$, ${\eta }_2=-{10}^6$
PF-LEX 3	$\epsilon =\delta =T^{-1/3}$

Table 4. Table 5 : Priority-based and priority-free regrets of OM-LEX 2, NOM-LEX 4 and 5 in Settings 4 and 5.

	Setting 4									Setting 5
Learners	Obj. 1			Obj. 2			Obj. 3			Obj. 1			Obj. 2			Obj. 3
OM-LEX 2
pr.-based	2000	$\pm$	100	821	$\pm$	75	367	$\pm$	59	1010	$\pm$	82				373	$\pm$	72
pr.-free	1990	$\pm$	120	1440	$\pm$	110	1290	$\pm$	110	1040	$\pm$	81	-350	$\pm$	17	-350	$\pm$	17
NOM-LEX 4
pr.-based	6620	$\pm$	2000	2160	$\pm$	800	684	$\pm$	420	7180	$\pm$	2000				1160	$\pm$	470
pr.-free	6840	$\pm$	1800	-4490	$\pm$	3000	-7910	$\pm$	3200	7250	$\pm$	2200	-14100	$\pm$	4300	-5930	$\pm$	2500
NOM-LEX 5
pr.-based										6570	$\pm$	2700				1020	$\pm$	550
pr.-free										6670	$\pm$	2600	-12900	$\pm$	5200	-5860	$\pm$	2800

Table 5. Table 6 : List of system notations.

Notation	Definition	Description
$A$		Number of arms
$𝒜$	$\left[A\right]$	Set of all arms
$a\left(t\right)$		Selected arm in round $t$
$D$		Number of objectives
$𝒟$	$\left[D\right]$	Set of all objectives
${\mu }_a^i$		Expected reward of arm $a$ in objective $i$
${𝝁}_a$	$({\mu }_a^1,\dots ,{\mu }_a^D)$	Expected reward vector of arm $a$
${\kappa }^i\left(t\right)$		Noise in objective $i$ in round $t$
$r^i\left(t\right)$	${\mu }_{a\left(t\right)}^i+{\kappa }^i\left(t\right)$	Reward of the selected arm in round $t$
${\succ }_{\mathrm{lex},i}$	$𝝁{\succ }_{\mathrm{lex},i}{𝝁}^{\prime }\iff {\mu }^j>{\mu }^{\prime j}$, $j=\min \left\{k\le i:{\mu }^k\ne {\mu }^{\prime k}\right\}$	Symbol for lexicographic dominance in the first $i$ objectives
${𝒜}_{\ast }^i$	$\{a:𝒜:{𝝁}_{a^{\prime }}{\nsucc }_{\mathrm{lex},i}{𝝁}_a,\forall a^{\prime }\in 𝒜\}$	Set of lexicographic optimal arms in the first $i$ objectives
${𝒜}_{\ast }$	${𝒜}_{\ast }^D$	Set of lexicographic optimal arms in all objectives
$\ast$	$\ast :\ast \in {𝒜}_{\ast }$	A lexicographic optimal arm
${\Delta }_a^i$	${\mu }_{\ast }^i-{\mu }_a^i$	Gap of arm $a$ in objective $i$
${\nabla }_a^i$	$\left|{\mu }_{\ast }^i-{\mu }_a^i\right|$	Absolute gap of arm $a$ in objective $i$
${𝒮}_{\ast }^i$	${𝒜}_{\ast }^{i-1}-{𝒜}_{\ast }^i$	Set of arms that are lexicographic optimal in the first $i-1$ objectives but not lexicographic optimal in the first $i$ objectives
${𝒮}^i$	$\{a:{\Delta }_a^i>0\}$	Set of suboptimal arms in objective $i$.
${\Delta }_{\max }^i$	${\max }_{a\in 𝒜}{\Delta }_a^i$	Maximum suboptimality gap in objective $i$
${\nabla }_a^{\max }$	${\max }_{i\in 𝒟}{\nabla }_a^i$	Maximum absolute gap of arm $a$
${\mathrm{𝐑𝐞𝐠}}_{pb}\left(T\right)$	$({\mathrm{Reg}}_{pb}^1\left(T\right),\dots ,{\mathrm{Reg}}_{pb}^D\left(T\right))$	Lexicographic priority-based regret
${\mathrm{𝐑𝐞𝐠}}_{pf}\left(T\right)$	$({\mathrm{Reg}}_{pf}^1\left(T\right),\dots ,{\mathrm{Reg}}_{pf}^D\left(T\right))$	Lexicographic priority-free regret
${\mathrm{Reg}}_{pb}^i\left(T\right)$	${\sum }_{t=1}^T{\Delta }_{a\left(t\right)}^i𝕀\left\{a\left(t\right)\in {𝒮}_{\ast }^i\right\}$	Priority-based regret in objective $i$
${\mathrm{Reg}}_{pf}^i\left(T\right)$	${\sum }_{t=1}^T{\Delta }_{a\left(t\right)}^i$	Priority-free regret in objective $i$
${\Delta }_{\min }^i$	${\min }_{a\in {𝒮}_{\ast }^i}{\Delta }_a^i$ for ${\mathrm{𝐑𝐞𝐠}}_{pb}\left(T\right)$, ${\min }_{a\in {𝒮}^i}{\Delta }_a^i$ for ${\mathrm{𝐑𝐞𝐠}}_{pf}\left(T\right)$	Minimum suboptimality gap in objective $i$

Table 6. Table 7 : List of notations for Case 1.

Notation	Definition	Description
$N_a\left(t\right)$	${\sum }_{t^{\prime }=1}^{t-1}𝕀\left\{a\left(t^{\prime }\right)=a\right\}$	Number of times arm $a$ was selected by the beginning of round $t$
${\hat{\mu }}_a^i\left(t\right)$	${\sum }_{t^{\prime }=1}^{t-1}{r}^i\left(t^{\prime }\right)/N_a\left(t\right)$	Sample mean of the rewards of arm $a$ in objective $i$ at the beginning of round $t$
${\widehat{𝒜}}_{\ast }\left(t\right)$		Set of estimated lexicographic optimal arms at the beginning of round $t$
$†\left(a\right)$	${\mathrm{argmax}}_{i\in 𝒟}{\nabla }_a^i$	Objective for which ${\nabla }_a^{†\left(a\right)}={\nabla }_a^{\max }$
${(\cdot )}_a^{†}$	${(\cdot )}_a^{†\left(a\right)}$

Table 7. Table 8 : List of notations for Case 2.

Notation	Definition	Description
${\eta }_i$		Near-lexicographic optimal expected reward in objective $i$
${\delta }_i$	${\mu }_{\ast }^i-{\eta }_i$	Gap between near-lexicographic optimal expected reward and the lexicographic optimal expected reward in objective $i$
$N_a\left(t\right)$	${\sum }_{t^{\prime }=1}^{t-1}𝕀\left\{a\left(t^{\prime }\right)=a\right\}$	Number of times arm $a$ was selected by the beginning of round $t$
${\hat{\mu }}_a^i\left(t\right)$	${\sum }_{t^{\prime }=1}^{t-1}{r}^i\left(t^{\prime }\right)/N_a\left(t\right)$	Sample mean of the rewards of arm $a$ in objective $i$ at the beginning of round $t$
${\widehat{𝒜}}_{\ast }\left(t\right)$	$\{a\in 𝒜:\forall i\in 𝒟,{\hat{\mu }}_a^i\left(t\right)>-\sqrt{4\log N_a\left(t\right)/N_a\left(t\right)}\}$	Set of estimated lexicographic optimal arms at the beginning of round $t$
$†\left(a\right)$	${\mathrm{argmax}}_{i\in 𝒟}\left({\Delta }_a^i-{\delta }_i\right)$
${(\cdot )}_a^{†}$	${(\cdot )}_a^{†\left(a\right)}$

Table 8. Table 9 : List of notations for Case 3.

Notation	Definition	Description
$\epsilon$		Suboptimality that is aimed to be tolerated
$\delta$		Confidence term
$N_a\left(t\right)$	${\sum }_{t^{\prime }=1}^{t-1}𝕀\left\{a\left(t^{\prime }\right)=a\right\}$	Number of times arm $a$ was selected by the beginning of round $t$
${\hat{\mu }}_a^i\left(t\right)$	${\sum }_{t^{\prime }=1}^{t-1}{r}^i\left(t^{\prime }\right)/N_a\left(t\right)$	Sample mean of the rewards of arm $a$ in objective $i$ at the beginning of round $t$
$c_a\left(t\right)$	See Section 4	Half of the length of the confidence interval of arm $a$ at the beginning of round $t$
$u_a^i\left(t\right)$	${\hat{\mu }}_a^i\left(t\right)+c_a\left(t\right)$	Upper confidence bound of arm $a$ in objective $i$ at the beginning of round $t$
$l_a^i\left(t\right)$	${\hat{\mu }}_a^i\left(t\right)-c_a\left(t\right)$	Lower confidence bound of arm $a$ in objective $i$ at the beginning of round $t$
$C_{i,t}$	See Section 4	Chains to in objective $i$ in round $t$
${\hat{a}}_{\ast }^i\left(t\right)$	${\mathrm{argmax}}_{a\in {\widehat{𝒜}}_{\ast }^{i-1}\left(t\right)}{u}_a^i\left(t\right)$	Optimistic near-lexicographic optimal arm in objective $i$ at the beginning of round $t$
${\widehat{𝒜}}_{\ast }^i\left(t\right)$	$\{a\in 𝒜:a{C}_{i,t}{\hat{a}}_{\ast }^i\left(t\right)\}$	Set of estimated lexicographic optimal arms in the first $i$ objectives at the beginning of round $t$

Equations

$${\hat{\mathcal{A}}}_{\ast }(t):=\left\{a\in \mathcal{A}:\forall i\in \mathcal{D},|{\hat{\mu }}_a^i(t)|<\sqrt{\frac{4\log N_a(t)}{N_a(t)}}\right\}.$$

$$\mathbb{E}[{\mathrm{Reg}}_{pb}^i(T)]\le \sum _{a\in {\mathcal{S}}_{\ast }^i}\left(\left(\frac{{\pi }^2}{3}D+1\right){\Delta }_a^i+\frac{36}{{\nabla }_a^{\max }}\log \frac{17}{{\nabla }_a^{\max }}\right).$$

$${\hat{\mathcal{A}}}_{\ast }(t):=\left\{a\in \mathcal{A}:\forall i\in \mathcal{D},{\hat{\mu }}_a^i(t)>-\sqrt{\frac{4\log N_a(t)}{N_a(t)}}\right\}.$$

$$\mathbb{E}[{\mathrm{Reg}}_{pb}^i(T)]\le \sum _{a\in {\mathcal{S}}_{\ast }^i}\left(\left(\frac{{\pi }^2}{6}D+1\right){\Delta }_a^i+\frac{36{\Delta }_a^i}{({\max }_{j\in \mathcal{D}}({\Delta }_a^j-{\delta }_j){)}^2}\log \frac{17}{{\max }_{j\in \mathcal{D}}({\Delta }_a^j-{\delta }_j)}\right).$$

$$c_a(t):=\sqrt{\frac{1+N_a(t)}{N_a^2(t)}\left(1+2\log \left(\frac{AD\sqrt{1+N_a(t)}}{\delta }\right)\right)}$$

$${\mathrm{Reg}}_{pb}^i(T)\le 4\sqrt{2}{B}_{T,\delta }\sqrt{|{\mathcal{S}}_{\ast }^i|T}+\left(3+\frac{16}{{\epsilon }^2}\log \frac{2\sqrt{e}AD}{\epsilon \delta }\right)|{\mathcal{S}}_{\ast }^i|{\Delta }_{\max }^i+\epsilon (A-1)T$$

$${\mathrm{Reg}}_{pb}^i(T)\le 4\sqrt{2}{B}_{T,\delta }\sqrt{|{\mathcal{S}}_{\ast }^i|T}+\left(3+16T^{2/3}\log \frac{2\sqrt{e}AD{T}^{1/3}}{\delta }\right)|{\mathcal{S}}_{\ast }^i|{\Delta }_{\max }^i+(A-1)T^{2/3}.$$

$$\mathbb{E}[{\mathrm{Reg}}_{pf}^i(T)]\le \sum _{a\in {\mathcal{S}}^i}\left(\left(\frac{{\pi }^2}{3}D+1\right){\Delta }_a^i+\frac{36}{{\nabla }_a^{\max }}\log \frac{17}{{\nabla }_a^{\max }}\right).$$

$$\mathbb{E}[{\mathrm{Reg}}_{pf}^i(T)]$$

$$\le \mathbb{E}\left[\sum _{t=1}^T{\Delta }_{a(t)}^i\mathbb{I}\{a(t)\in {\mathcal{S}}^i\}\right]$$

$$\le \sum _{a\in {\mathcal{S}}^i}\left(\left(\frac{{\pi }^2}{3}D+1\right){\Delta }_a^i+\frac{36}{{\nabla }_a^{\max }}\log \frac{17}{{\nabla }_a^{\max }}\right)$$

$$\mathbb{E}[{\mathrm{Reg}}_{pf}^i(T)]\le \sum _{a\in {\mathcal{S}}^i}\left(\left(\frac{{\pi }^2}{6}D+1\right){\Delta }_a^i+\frac{36{\Delta }_a^i}{({\max }_{j\in \mathcal{D}}({\Delta }_a^j-{\delta }_j){)}^2}\log \frac{17}{{\max }_{j\in \mathcal{D}}({\Delta }_a^j-{\delta }_j)}\right).$$

$$\mathbb{E}[{\mathrm{Reg}}_{pf}^i(T)]$$

$$\le \mathbb{E}\left[\sum _{t=1}^T{\Delta }_{a(t)}^i\mathbb{I}\{a(t)\in {\mathcal{S}}^i\}\right]$$

$$\le \sum _{a\in {\mathcal{S}}^i}\left(\left(\frac{{\pi }^2}{6}D+1\right){\Delta }_a^i+\frac{36{\Delta }_a^i}{({\max }_{j\in \mathcal{D}}({\Delta }_a^j-{\delta }_j){)}^2}\log \frac{17}{{\max }_{j\in \mathcal{D}}({\Delta }_a^j-{\delta }_j)}\right)$$

$$\mathbb{E}[{\mathrm{Reg}}_s^i(T)]\le \sum _{a\in {\mathcal{S}}_s^i}\left(\left(\frac{{\pi }^2}{6}D+1\right)({\Delta }_a^i-{\delta }_i)+\frac{36}{{\max }_{j\in \mathcal{D}}({\Delta }_a^j-{\delta }_j)}\log \frac{17}{{\max }_{j\in \mathcal{D}}({\Delta }_a^j-{\delta }_j)}\right).$$

$$\frac{{\Delta }_a^i-{\delta }_i}{({\max }_{j\in \mathcal{D}}({\Delta }_a^j-{\delta }_j){)}^2}\le \frac{1}{{\max }_{j\in \mathcal{D}}({\Delta }_a^j-{\delta }_j)}$$

$$\mathbb{E}[{\mathrm{Reg}}_s(T)]\le \sum _{a\in {\mathcal{S}}_s}\left(\left(\frac{{\pi }^2}{6}D+1\right)({\Delta }_a-\delta )+\frac{36}{{\Delta }_a-\delta }\log \frac{17}{{\Delta }_a-\delta }\right).$$

$$\sum _{w=1}^{\infty }\mathbb{P}\left({\hat{\mu }}_a^i(t)-{\mu }_a^i>\Delta -\sqrt{\frac{4\log N_a(t)}{N_a(t)}}|N_a(t)=w\right)$$

$$=\sum _{w=1}^{\infty }\mathbb{P}\left({\hat{\mu }}_a^i(t)-{\mu }_a^i<\sqrt{\frac{4\log N_a(t)}{N_a(t)}}-\Delta |N_a(t)=w\right)$$

$$\le \frac{36}{{\Delta }^2}\log \frac{17}{\Delta }.$$

$$\mathbb{E}[{\mathrm{Reg}}^i(T)]$$

$$=\mathbb{E}\left[\sum _{a\in {\mathcal{S}}_{\ast }^i}\sum _{t=1}^T\mathbb{I}\{a(t)=a\}{\Delta }_a^i\right]$$

$$=\mathbb{E}\left[\sum _{a\in {\mathcal{S}}_{\ast }^i}\sum _{t=1}^T\mathbb{I}\{t\le A,a(t)=a\}{\Delta }_a^i\right]$$

$$+\mathbb{E}\left[\sum _{a\in {\mathcal{S}}_{\ast }^i}\sum _{t=1}^T\mathbb{I}\left\{t>A,|{\hat{\mu }}_a^{†}(t)|<\sqrt{\frac{4\log N_a(t)}{N_a(t)}},a(t)=a\right\}{\Delta }_a^i\right]$$

$$+\mathbb{E}\left[\sum _{a\in {\mathcal{S}}_{\ast }^i}\sum _{t=1}^T\mathbb{I}\left\{t>A,|{\hat{\mu }}_a^{†}(t)|\ge \sqrt{\frac{4\log N_a(t)}{N_a(t)}},a(t)=a\right\}{\Delta }_a^i\right].$$

$$\mathbb{E}\left[\sum _{a\in {\mathcal{S}}_{\ast }^i}\sum _{t=1}^T\mathbb{I}\{t\le A,a(t)=a\}{\Delta }_a^i\right]$$

$$=\sum _{a\in {\mathcal{S}}_{\ast }^i}\mathbb{E}\left[\sum _{t=1}^T\mathbb{I}\{t=a\}\right]{\Delta }_a^i$$

$$=\sum _{a\in {\mathcal{S}}_{\ast }^i}{\Delta }_a^i.$$

$$\mathbb{E}\left[\sum _{a\in {\mathcal{S}}_{\ast }^i}\sum _{t=1}^T\mathbb{I}\left\{t>A,|{\hat{\mu }}_a^{†}(t)|<\sqrt{\frac{4\log N_a(t)}{N_a(t)}},a(t)=a\right\}{\Delta }_a^i\right]$$

$$=\sum _{a\in {\mathcal{S}}_{\ast }^i}\mathbb{E}\left[\sum _{w=1}^{w_a(T)-1}\sum _{t={\tau }_a(w)+1}^{{\tau }_a(w+1)}\mathbb{I}\left\{|{\hat{\mu }}_a^{†}(t)|<\sqrt{\frac{4\log N_a(t)}{N_a(t)}},a(t)=a\right\}\right]{\Delta }_a^i$$

$$=\sum _{a\in {\mathcal{S}}_{\ast }^i}\mathbb{E}\left[\sum _{w=1}^{w_a(T)-1}\mathbb{I}\left\{|{\hat{\mu }}_a^{†}({\tau }_a(w+1))|<\sqrt{\frac{4\log w}{w}}\right\}\right]{\Delta }_a^i$$

$$\le \sum _{a\in {\mathcal{S}}_{\ast }^i}\sum _{w=1}^{\infty }\mathbb{P}\left(|{\hat{\mu }}_a^{†}({\tau }_a(w+1))|<\sqrt{\frac{4\log w}{w}}\right){\Delta }_a^i.$$

$$\sum _{w=1}^{\infty }\mathbb{P}\left(|{\hat{\mu }}_a^{†}({\tau }_a(w+1))|<\sqrt{\frac{4\log w}{w}}\right)$$

$$\le \sum _{w=1}^{\infty }\mathbb{P}\left({\hat{\mu }}_a^{†}({\tau }_a(w+1))-{\mu }_a^{†}<\sqrt{\frac{4\log w}{w}}-{\nabla }_a^{†}\right)$$

$$\le \frac{36}{({\nabla }_a^{†}{)}^2}\log \frac{17}{{\nabla }_a^{†}},$$

$$\sum _{w=1}^{\infty }\mathbb{P}\left(|{\hat{\mu }}_a^{†}({\tau }_a(w+1))|<\sqrt{\frac{4\log w}{w}}\right)$$

$$\le \sum _{w=1}^{\infty }\mathbb{P}\left({\hat{\mu }}_a^{†}({\tau }_a(w+1))-{\mu }_a^{†}>{\nabla }_a^{†}-\sqrt{\frac{4\log w}{w}}\right)$$

$$\le \frac{36}{({\nabla }_a^{†}{)}^2}\log \frac{17}{{\nabla }_a^{†}},$$

$$\text{eqref}eqn:case1-decomp2c$$

Full text

1 Introduction

A vast number of decision making and learning tasks involve multidimensional performance metrics (objectives). Examples include recommending items in a recommender system to optimize accuracy, diversity and novelty [1, 2], learning lexicographic optimal routing flows in wireless networks [3], and adjusting the dose of radiation therapy for cancer patients while prioritizing target coverage over proximity of the therapy to the organs at risk [4]. In the aforementioned problems, the learner aims to choose arms that yield high rewards in all of the objectives; however, it prefers arms that yield high rewards in the low priority objectives only if they do not compromise the rewards in the high priority objectives.

Motivated by the above applications, in this paper we propose a new multiarmed bandit (MAB) problem called the lexicographic multiarmed bandit (Lex-MAB). In this problem, the learner’s priority over the objectives is formally captured by lexicographic ordering. Essentially, given $D$ objectives indexed by the set ${\cal D}:=[D]$, objective $i$ has a higher priority than objective $j$ if $i<j$.111For a positive integer $D$, $[D]:=\{1,\ldots,D\}$ This priority induces a preference over the finite set of arms denoted by ${\cal A}$. Namely, given two arms $a$ and $a^{\prime}$ with the corresponding real-valued expected reward vectors $\bm{\mu}_{a}:=(\mu^{1}_{a},\ldots,\mu^{D}_{a})$ and $\bm{\mu}_{a^{\prime}}:=(\mu^{1}_{a^{\prime}},\ldots,\mu^{D}_{a^{\prime}})$, we say that arm $a$ lexicographically dominates arm $a^{\prime}$ in the first $i\leq D$ objectives (written as $a\succ_{\mathrm{lex},i}a^{\prime}$) if $\mu^{j}_{a}>\mu^{j}_{a^{\prime}}$, where $j:=\min\{k\leq i:\mu^{k}_{a}\neq\mu^{k}_{a^{\prime}}\}$.222If there is no such $j$, then $\mu^{k}_{a}=\mu^{k}_{a^{\prime}}$ for all $k\in[i]$, which implies that $\bm{\mu}_{a}$ does not lexicographically dominate $\bm{\mu}_{a^{\prime}}$ in the first $i$ objectives. Here, the latter expression is succinctly expressed as $\bm{\mu}_{a}\succ_{\mathrm{lex},i}\bm{\mu}_{a^{\prime}}$. Based on this preference, the set of lexicographic optimal arms are defined as the ones that are not lexicographically dominated by any other arm in all of the $D$ objectives, which is given as ${\cal A}_{*}:=\{a\in\mathcal{A}:\bm{\mu}_{a^{\prime}}\nsucc_{\mathrm{lex},D}\bm{\mu}_{a},\forall a^{\prime}\in\mathcal{A}\}$.

In the Lex-MAB, in each round $t$ the learner selects an arm $a(t)=a$ from ${\cal A}$, and then, receives a $D$-dimensional random reward vector $\bm{r}(t)=(r^{1}(t),\ldots,r^{D}(t))$ that is drawn from a fixed distribution with expectation vector $\bm{\mu}_{a}$. The goal of the learner is to perform as well as an oracle which perfectly knows the set of lexicographic optimal arms and selects a lexicographic optimal arm in every round. We capture the ordering of the objectives by introducing a multidimensional regret measure called the lexicographic regret. As this regret notion is fundamentally different from the scalar regret notion used in the classical stochastic MAB [5], minimizing it requires exploiting the multidimensional nature of the rewards and ordering of the objectives both in algorithm design and technical analysis.

This is a challenging task because simple techniques such as turning the problem into an MAB with scalar rewards by using scalarizing methods from multiobjective optimization [6] will not work since the solution of the scalarized problem may not produce lexicographic optimal arms. The problem is further complicated due to the fact that without any prior knowledge on the expected arm rewards, it is impossible to identify lexicographic optimal arms with high probability. This can be observed by considering a problem instance with arms $a$ and $b$, and $D=2$, such that $a$ and $b$ have the same expected reward in objective $1$ and $a$ is the only lexicographic optimal arm. Although in this problem the learner can identify with high probability which arm is better in the second objective, it can never be sure about the lexicographic optimality of this arm. We call this problem the identifiability problem.333It also exists in the best arm identification problem with a fixed confidence (see Chapter 33 in LS19bandit-book [7]).

The challenges described above motivates us to focus on the cases when the learner has prior knowledge on expected rewards. Specifically, we consider two types of prior knowledge, which generalize the prior knowledge introduced in bubeck2013bounded [8] and vakili2013achieving [9] to multidimensional rewards. In the first case, we assume that the expected rewards of a lexicographic optimal arm are known. In the second case, we assume that near-lexicographic optimal expected rewards are known. Then, we build learning algorithms that utilize the prior information to achieve uniformly bounded in time lexicographic regret for both cases.

Importantly, for the first case, we show that the regret in each objective due to selecting a suboptimal arm $a$ is inversely proportional to the maximum of the gaps of arm $a$ over all objectives. This shows that having prior information over multiple objectives speeds up elimination of suboptimal arms. This is analogous to the combinatorial MAB [10] in the sense that observations from one objective can help ruling out suboptimal arms in other objectives. We also prove that a similar gain appears in the second case, albeit we cannot rule out an arm performing much better than a lexicographic optimal arm in some objective as suboptimal.

In addition to the cases with prior information, as the third case, we consider learning without any prior information. In this setting, despite the identifiability problem, we show that it is possible to achieve $\tilde{O}(T^{2/3})$ gap-free regret by learning to select near-lexicographic optimal arms. Finally, we numerically evaluate the performance of the learner with and without prior information on several multiobjective learning problems with lexicographically ordered objectives.

Rest of the paper is organized as follows. Related work is given in Section 2. The Lex-MAB, the lexicographic regret and two types of prior information are defined in Section 3. Algorithms and regret bounds for the Lex-MAB under prior information and for the prior-free case are considered in Section 4. Experimental results are given in Section 5 followed by the concluding remarks in Section 6. Proofs, additional simulations, tables of notation and the code for experimental results are given in the appendix.

2 Related Work

lai1985asymptotically [5] shows that in the classical stochastic MAB problem, for any uniformly good policy the regret grows at least logarithmically over time. As opposed to this, lai1984optimal [11] proves for the two-armed stochastic bandit that when the learner has prior information on the maximum expected reward $\mu^{*}$ and the minimum nonzero suboptimality gap $\Delta$, there exists policies that can achieve uniformly bounded regret. This idea is further investigated in bubeck2013bounded [8], which shows that bounded regret of order $1/\Delta$ is achieved for the case with finitely many arms when the learner knows $\mu^{*}$ and a positive lower bound on $\Delta$. garivier [12] studies the case where only $\mu^{*}$ is known and proposes an algorithm with bounded regret of order $\log(1/\Delta)(1/\Delta)$, and also proves a lower bound of order $1/\Delta$. This paper also provides a generic tool to prove gap-dependent lower bounds on the regret. Similarly, bubeck2013prior [13] considers Thompson sampling and shows that its regret is uniformly bounded when $\mu^{*}$ and a positive lower bound on $\Delta$ are known. On the other hand, vakili2013achieving [9] considers a weaker prior information model where the learner knows a near-optimal expected reward $\eta$, which can be computed using $\mu^{*}$ and a positive lower bound on $\Delta$. The proposed algorithm obtains $\sum_{a}\Delta_{a}/\delta^{3}$ regret, where $\delta=\mu^{*}-\eta<\Delta$ and $\Delta_{a}$ is the suboptimality gap of arm $a$. mersereau2009structured [14] and lattimore2014bounded [15] consider as prior information the knowledge of parametrized expected reward functions for each arm. In these works, the only unknown is the true parameter, which can be estimated by using reward observations from all of the arms.

Different from the works mentioned above, in this paper we consider a multiobjective MAB problem with lexicographically ordered objectives. We design algorithms that exploit the prior information in all objectives simultaneously to rule out arms that are not lexicographic optimal. Our regret bounds match the one in [12] and improve the one in [9] for the case with a single objective.

Numerous works have investigated regret minimization in multiobjective variants of the MAB problem. For instance, drugan2013designing [16] defines for each suboptimal arm its distance to the Pareto front as the Pareto suboptimality gap and the regret as the sum of the Pareto suboptimality gaps of the arms chosen by the learner. It proposes a learning algorithm that achieves $O(\log T)$ gap dependent Pareto regret. turgay2018multi [17] considers the multiobjective contextual MAB problem with similarity information, and extends the contextual zooming algorithm in slivkins2014contextual [18] to minimize the Pareto regret while making fair selections among the estimated Pareto optimal arms. The proposed algorithm is shown to achieve $\tilde{O}(T^{(1+d_{p})/(2+d_{p})})$ Pareto regret where $d_{p}$ is the Pareto zooming dimension. In addition, tekin2018multi [19] considers a biobjective contextual MAB problem with lexicographically ordered objectives. Unlike this work, we study the general case with $D$ lexicographically ordered objectives and focus on the importance of prior information in learning.

Our problem is also related to the tresholding MAB studied in locatelli2016optimal [20] and satisficing MAB studied in reverdy2017satisficing [21]. Specifically, if the prior information is given with respect to a target arm which is not necessarily a lexicographic optimal arm, and the regret in each objective is defined with respect to the expected reward of the target arm in that objective, then our algorithms for the cases with prior information can easily be adapted to minimize the regret with respect to the target arm. Specifically, we adapt our second case to the satisfaction-in-mean-reward problem introduced in [21] and prove that our algorithm achieves bounded regret for this problem as well.444See Section A.2 for our results on the multiarmed bandit with satisficing objectives.

3 Problem Formulation

System Model: We consider rounds indexed by $t\in\{1,2,\ldots\}$. In each round $t$, the learner first selects an arm $a(t)$ from the finite arm set $\mathcal{A}:=[A]$, and then, observes a random reward for each objective $i\in\mathcal{D}:=[D]$, denoted by $r^{i}(t)$, which is equal to $\mu_{a(t)}^{i}+\kappa^{i}(t)$, where $\mu_{a}^{i}$ denotes the expected reward of arm $a$ in objective $i$ and $\kappa^{i}(t)$ denotes the zero mean noise. The learner does not know the expected reward vector $\bm{\mu}_{a}:=(\mu_{a}^{1},\ldots,\mu_{a}^{D})$, $a\in\mathcal{A}$ beforehand, and given $a(t)=a$, the noise vector $\{\kappa^{1}(t),\ldots,\kappa^{D}(t)\}$ is sampled from a fixed (unknown) multivariate distribution $\bm{\nu}_{a}$, independent of the other rounds. Moreover, its marginal distribution is 1-sub-Gaussian, i.e., $\forall a\in{\cal A}$ and $\forall\lambda\in\mathbb{R},\mathbb{E}[e^{\lambda\kappa^{i}(t)}|a(t)=a]\leq\exp(\lambda^{2}/2)$.555Noise can be dependent over the objectives. The assumption on the noise distribution is very general as it covers the Gaussian distribution with zero mean and unit variance, and any bounded zero mean distribution defined over an interval of length 2.

Lexicographic Optimality: For two $D$-dimensional real-valued vectors $\bm{\mu}:=(\mu^{1},\ldots,\mu^{D})$ and $\bm{\mu^{\prime}}:=(\mu^{\prime 1},\ldots,\mu^{\prime D})$, and $i\in[D]$, we say that $\bm{\mu}$ lexicographically dominates $\bm{\mu^{\prime}}$ in the first $i$ objectives, denoted by $\bm{\mu}\succ_{\mathrm{lex},i}\bm{\mu^{\prime}}$, if $\mu^{j}>\mu^{\prime j}$, where $j:=\min\{k\leq i:\mu^{k}\neq\mu^{\prime k}\}$. Based on this, we say that arm $a$ lexicographically dominates arm $a^{\prime}$ in the first $i$ objectives if $\bm{\mu}_{a}\succ_{\mathrm{lex},i}\bm{\mu}_{a^{\prime}}$. The complement of this is denoted by $\bm{\mu}_{a}\nsucc_{\mathrm{lex},i}\bm{\mu}_{a^{\prime}}$.

Let $\mathcal{A}_{*}^{i}:=\{a\in\mathcal{A}:\bm{\mu}_{a^{\prime}}\nsucc_{\mathrm{lex},i}\bm{\mu}_{a},\forall a^{\prime}\in\mathcal{A}\}$ denote the set of lexicographic optimal arms in the first $i$ objectives and define ${\cal A}_{*}:=\mathcal{A}_{*}^{D}$. Clearly, we have $\mathcal{A}_{*}^{i+1}\subseteq\mathcal{A}_{*}^{i}$ for $i\in[D-1]$. We use $*$ to denote an arm that is lexicographic optimal in all objectives, and $\mu_{*}^{i}$ to denote the expected reward of this arm in objective $i$. Moreover, we define the gap of arm $a$ in objective $i$ as $\Delta_{a}^{i}:=\mu_{*}^{i}-\mu_{a}^{i}$ and the absolute gap of arm $a$ in objective $i$ as $\nabla_{a}^{i}:=|\mu_{*}^{i}-\mu_{a}^{i}|$. For $i\in\{2,\ldots,D\}$, we let $\mathcal{S}_{*}^{i}:=\mathcal{A}_{*}^{i-1}-\mathcal{A}_{*}^{i}$ denote the set of arms that are lexicographic optimal in the first $i-1$ objectives but not lexicographic optimal in the first $i$ objectives and define $\mathcal{S}_{*}^{1}:=\mathcal{A}-\mathcal{A}_{*}^{1}$. Note that $a\in\mathcal{S}_{*}^{i}$ implies that $\Delta_{a}^{i}>0$. The set of suboptimal arms in objective $i$ is given as $\mathcal{S}^{i}:=\{a:\Delta_{a}^{i}>0\}$. We also define the maximum suboptimality gap in objective $i$ as $\Delta_{\max}^{i}:=\max_{a\in\mathcal{A}}\Delta_{a}^{i}$ and the maximum absolute gap as $\nabla_{a}^{\max}:=\max_{j\in\mathcal{D}}\nabla_{a}^{j}$. Finally, let $\Delta_{\min}^{i}:=\min_{a\in\mathcal{S}_{*}^{i}}\Delta_{a}^{i}$ denote the minimum suboptimality gap among arms in $\mathcal{S}_{*}^{i}$.666If $S_{*}^{i}=\text{\O}$, then $\Delta_{\min}^{i}=\infty$.

Types of Prior Knowledge:

(Case 1) Lexicographic optimal expected rewards are known: The learner knows $\mu_{*}^{i}$ for all $i\in\mathcal{D}$. For this case, we assume $\mu_{*}^{i}=0$, $\forall i\in\mathcal{D}$ without any loss of generality.

(Case 2) Near-lexicographic optimal expected rewards are known: The learner knows $\eta_{i}$ such that $\mu_{*}^{i}-\Delta_{\min}^{i}<\eta_{i}<\mu_{*}^{i}$ for all $i\in\mathcal{D}$. For this case, we define $\delta_{i}:=\mu_{*}^{i}-\eta_{i},\forall i\in\mathcal{D}$ and assume $\eta_{i}=0,\forall i\in\mathcal{D}$ without any loss of generality.

(Case 3) No prior knowledge on the expected rewards.

Remark 1**.**

In Case 1, if $\mu_{*}^{i}$s are not equal to [math], we can subtract them from the rewards to obtain $\tilde{r}^{i}(t):=r^{i}(t)-\mu_{*}^{i}$. Under these transformed rewards, we will have $\tilde{\mu}_{*}^{i}=0,\forall i\in\mathcal{D}$ and the gaps that we have defined will not be affected. Similarly, in Case 2, we can subtract $\eta_{i}$s from the rewards to obtain $\tilde{r}^{i}(t):=r^{i}(t)-\eta_{i}$.

Regret Definitions: The (pseudo) regret of the learner is measured with respect to an oracle, which knows the expected rewards of the arms, and chooses a lexicographic optimal arm in each round. We define two notions of regret: Priority-based and priority-free regrets in objective $i$ are given as $\mathrm{Reg}^{i}_{pb}(T):=\sum_{t=1}^{T}\Delta_{a(t)}^{i}\mathbb{I}\{a(t)\in\mathcal{S}_{*}^{i}\}$ and $\mathrm{Reg}^{i}_{pf}(T):=\sum_{t=1}^{T}\Delta_{a(t)}^{i}$ respectively. The lexicographic priority-based and priority-free regrets are defined as the tuples $\bm{\mathrm{Reg}}_{pb}(T):=(\mathrm{Reg}^{1}_{pb}(T),\ldots,\mathrm{Reg}^{D}_{pb}(T))$ and $\bm{\mathrm{Reg}}_{pf}(T):=(\mathrm{Reg}^{1}_{pf}(T),\ldots,\mathrm{Reg}^{D}_{pf}(T))$ respectively. Subscripts will be removed from the notation when the considered regret notion is clear from the context.

For $\bm{\mathrm{Reg}}_{pb}(T)$, when $a(t)\in\mathcal{S}_{*}^{i}$, regret is incurred only in objective $i$. No regret is incurred for $j<i$ since $\Delta_{a(t)}^{j}=0$. In addition, no regret is incurred for $j>i$ when $a(t)\in\mathcal{S}_{*}^{i}$. This definition of regret is consistent with the priority that the learner assigns to each objective. Since lexicographic ordering implies that even a small improvement in the expected reward in objective $i$ is more important than any improvement in the expected rewards of objectives $j>i$, the learner does not care about the loss it incurs in higher indexed objectives when $a(t)\in\mathcal{S}_{*}^{i}$. For $\bm{\mathrm{Reg}}_{pf}(T)$, an arm $a$ for which $\mu^{i}_{a}>\mu^{i}_{*}$ will incur negative regret in objective $i$, but positive regret in some other objective.

We say that the regret is $O(\max\{f_{1}(T),\ldots,f_{D}(T)\})$ when $\mathrm{Reg}^{i}(T)\in O(f_{i}(T))$ for $i\in\mathcal{D}$. Under both notions of regret, the (cumulative) regret of any arm selection strategy cannot lexicographically dominate the cumulative regret of always selecting a lexicographic optimal arm, which is essentially the zero vector. Therefore, the time-averaged expected rewards of any algorithm that achieves sublinear $\bm{\mathrm{Reg}}_{pb}(T)$ or $\bm{\mathrm{Reg}}_{pf}(T)$ will converge (as $T\rightarrow\infty$) to the lexicographic optimal expected rewards. In addition, under $\bm{\mathrm{Reg}}_{pf}(T)$ the lexicographic ordering between the cumulative expected rewards and the regrets of any pair of sequences of arms $(a(1),\ldots,a(T))$ and $(a^{\prime}(1),\ldots,a^{\prime}(T))$ will be the same.

Remark 2**.**

For Case 1, the bound for $\bm{\mathrm{Reg}}_{pb}(T)$ given in Theorem 1 also holds for $\bm{\mathrm{Reg}}_{pf}(T)$ if we replace $\mathcal{S}_{*}^{i}$ with $\mathcal{S}^{i}$. For Case 2, if we redefine the minimum suboptimality gap as $\Delta_{\min}^{i}:=\min_{a\in\mathcal{S}^{i}}\Delta_{a}^{i}$ (which implies a stronger prior knowledge on the near-lexicographic optimal expected rewards), and replace $\mathcal{S}_{*}^{i}$ with $\mathcal{S}^{i}$, then the bound for $\bm{\mathrm{Reg}}_{pb}(T)$ in Theorem 2 also holds for $\bm{\mathrm{Reg}}_{pf}(T)$. Theorems 4 and 5 in the appendix bound $\bm{\mathrm{Reg}}_{pf}(T)$ for Cases 1 and 2 respectively. On the other hand, for Case 3, the regret analysis only holds for $\bm{\mathrm{Reg}}_{pf}(T)$ (see Section 4 for the details).

4 Learning Algorithms and Regret Bounds

A Learning Algorithm for Case 1: We propose Optimal Mean based Lexicographic Exploration and eXploitation (OM-LEX) given in Algorithm 1 for the prior information described in Case 1. In essence, OM-LEX generalizes the arm selection rule proposed in Algorithm 1 in garivier [12] to multiple objectives. It keeps for each arm $a$ a counter $N_{a}$ that counts how many times arm $a$ was selected prior to the current round and the sample mean estimate $\hat{\mu}_{a}^{i}$ of the rewards from objective $i$ of arm $a$ observed prior to the current round for all $i\in{\cal D}$. The values of these variables at the beginning of round $t$ are denoted by $N_{a}(t)$ and $\hat{\mu}_{a}^{i}(t)$ respectively.

OM-LEX starts by selecting each arm exactly once. In the remaining rounds, it checks whether there exists an arm whose sample mean reward in objective $i$ is within a shrinking neighborhood of the lexicographic optimal arm’s expected reward for all objectives $i\in\mathcal{D}$. For this, it computes the set of estimated lexicographic optimal arms in round $t$ as

[TABLE]

If $\hat{{\cal A}}_{*}(t)\neq\text{\O}$, then OM-LEX exploits by selecting one of the arms in $\hat{{\cal A}}_{*}(t)$ uniformly at random as it expects only the lexicographic optimal arms to satisfy this condition in the long run. If no such arm exists, then OM-LEX explores by playing all arms in a round-robin fashion. The following theorem shows that the expected priority-based regret of OM-LEX is uniformly bounded in time.

Theorem 1**.**

When OM-LEX is run, $\forall i\in\mathcal{D}$ and $\forall T\geq 1$, we have

[TABLE]

When $D=1$, this result is identical to the regret bound in Theorem 9 in [12] except for some constants. In the multiobjective case, we see that the regret induced by an arm in one objective depends on the maximum of the absolute gaps of the same arm over all objectives. As long as the arm has a large absolute gap in at least one objective, it is easy to identify it as a suboptimal arm.

A Learning Algorithm for Case 2: We propose Near Optimal Mean based Lexicographic Exploration and eXploitation (NOM-LEX). NOM-LEX has almost the same structure with OM-LEX. Its pseudocode is exactly the same as Algorithm 1 except two differences: Firstly, its input prior knowledge (given in line 1 of Algorithm 1) is $\eta_{i}=0$, $\forall i\in{\cal D}$. Secondly, NOM-LEX computes the set of estimated lexicographic optimal arms in round $t$ (given in line 6 of Algorithm 1) as

[TABLE]

The next theorem bounds the expected priority-based regret of NOM-LEX.

Theorem 2**.**

When NOM-LEX is run, $\forall i\in\mathcal{D}$ and $\forall T\geq 1$, we have

[TABLE]

From Theorem 2, we see that the regret due to a suboptimal arm $a$ in objective $i$ depends on the maximum squared difference between the suboptimality gaps of that arm and near-lexicographic optimal expected rewards over all objectives. This also shows that the prior knowledge in other objectives may help the learner attain smaller regret in objective $i$. However, since the lexicographic optimal expected rewards are not known, unlike Case 1, we cannot rule out a suboptimal arm in objective $i$ by observing that it is much better than a lexicographic optimal arm in another objective.

A Learning Algorithm for Case 3: We propose Prior Free Lexicographic Exploration and eXploitation (PF-LEX) given in Algorithm 2, which learns to select near-lexicographic optimal arms without any prior information on the mean arm rewards. PF-LEX takes as input $\varepsilon>0$, which is proportional to the suboptimality that the learner aims to tolerate in the objectives (this will be adjusted based on the time horizon $T$). Similar to OMG-LEX, it keeps for each arm $a$ the counter $N_{a}(t)$ and the sample mean reward $\hat{\mu}^{i}_{a}(t)$, $\forall i\in\mathcal{D}$.

Arm selection of PF-LEX in round $t$ depends on the confidence intervals in the first $D-1$ objectives. The upper confidence bound (UCB) and the lower confidence bound (LCB) of arm $a$ in objective $i$ are given as $u_{a}^{i}(t):=\hat{\mu}_{a}^{i}(t)+c_{a}(t)$ and $l_{a}^{i}(t):=\hat{\mu}_{a}^{i}(t)-c_{a}(t)$ respectively. Here

[TABLE]

represents the uncertainty in arm $a$’s reward, and $\delta$ is called the confidence term, which is also given as input to PF-LEX. As expected, the uncertainty decreases as arm $a$ gets selected. It is easy to see that $\mu^{i}_{a}\in[l_{a}^{i}(t),u_{a}^{i}(t)]$ with high probability for all objectives and all rounds. In each round, PF-LEX estimates the set of near-lexicographic optimal arms. For this, similar to joseph2016fairness [22], we say that arms $a$ and $b$ are linked in objective $i$ if $[l_{a}^{i}(t),u_{a}^{i}(t)]\cap[l_{b}^{i}(t),u_{b}^{i}(t)]\neq\text{\O}$. When $a$ and $b$ are in the same component of the transitive closure of the linked relation in objective $i$, we say they are chained in objective $i$ and write $a\>C_{i,t}\>b$. Starting from $\hat{\mathcal{A}}_{*}^{0}(t)=\mathcal{A}$, PF-LEX recursively computes the estimate $\hat{{\cal A}}^{i}_{*}(t)$ of ${\cal A}^{i}_{*}$ for $i\in[D-1]$. After it computes $\hat{{\cal A}}_{*}^{i-1}(t)$, it identifies the optimistic near-lexicographic optimal arm in objective $i$ as $\hat{a}^{i}_{*}(t)=\operatorname{argmax}_{a\in\hat{{\cal A}}_{*}^{i-1}(t)}u_{a}^{i}(t)$. Then, it sets $\hat{{\cal A}}^{i}_{*}(t)=\{a\in\hat{{\cal A}}_{*}^{i-1}(t):a\>C_{i,t}\>\hat{a}_{*}^{i}(t)\}$.

Suppose we always select $\hat{a}_{*}^{D}(t)$, which happens to be in ${\cal S}^{i}_{*}$ for some round $t$. For this case, we show in Lemma 3 in the appendix that the regret incurred in this round is bounded by the length of the chain formed by $\hat{\mathcal{A}}_{*}^{i}(t)$. In order to guarantee regret that is proportional to $\varepsilon$, we want the length of the chains not be more than a constant factor of $\varepsilon$. As it is not always possible to shrink the chains by always selecting $\hat{a}_{*}^{D}(t)$, to achieve our goal, we require all arms in $\hat{\mathcal{A}}_{*}^{1}(t)$ to have narrow confidence intervals. Thus, PF-LEX selects $a(t)=a\in\hat{\mathcal{A}}_{*}^{1}(t)$ if there is an arm $a$ with high uncertainty, i.e., $c_{a}(t)>\varepsilon/2$. On the other hand, if $c_{a}(t)\leq\varepsilon/2$ for all $a\in\hat{\mathcal{A}}_{*}^{1}(t)$, then PF-LEX simply selects $a(t)=\hat{a}^{D}_{*}(t)$. Algorithm 2 shows a more efficient implementation of PF-LEX that does not compute $\hat{{\cal A}}_{*}^{j}(t)$ for $j>1$ when $a(t)$ is selected from $\hat{{\cal A}}_{*}^{1}(t)$. Finally, After PF-LEX selects arm $a(t)$, it observes the random reward vector $\bm{r}(t)=(r^{1}(t),\ldots,\allowbreak r^{D}(t))$ of arm $a(t)$, and then, updates the sample mean estimates of the rewards in objectives $i\in\mathcal{D}$ and the counter of $a(t)$.

The following theorem shows that PF-LEX achieves $\tilde{O}(T^{2/3})$ regret.

Theorem 3**.**

When PF-LEX is run with $\delta\in(0,1)$ and $\varepsilon>0$, with probability at least $1-\delta$, for all $i\in\mathcal{D}$ and for all $T\geq 1$, we have

[TABLE]

where $B_{T,\delta}:=\sqrt{1+2\log(AD\sqrt{T}/\delta)}$. Given a particular time horizon $T$, by setting $\varepsilon=T^{-1/3}$, with probability at least $1-\delta$, we have

[TABLE]

Moreover, taking $\delta=1/T$, $\mathbb{E}[\mathrm{Reg}_{pb}^{i}(T)]=\tilde{O}(T^{2/3})$.

Remark 3**.**

Unlike the cases with prior information, an analogue of the regret bound in Theorem 3 will not hold for the priority-free regret when $\mathcal{S}^{i}_{*}$ is replaced by ${\cal S}^{i}$. Any two arms that are both lexicographic optimal in the first $i-1$ objectives are linked in these objectives with high probability. If one happens to be the selected arm, we are confident that they are both in $\hat{\mathcal{A}}_{*}^{i-1}$. When the selected arm is in $\mathcal{S}_{*}^{i}$, we use this fact and compare it to a lexicographic optimal arm to conclude that the gap of the selected arm in objective $i$ is smaller than the regret that we aim to tolerate. However, we fail to make any deductions about the higher indexed objectives.

5 Experiments

We demonstrate our results in three different settings with $A=3$ and $D=2$. All rewards are assumed to come from independent Bernoulli distributions in both objectives with expected reward vectors given in Table 2. In all settings, the only lexicographic optimal arm is the first arm and $\Delta_{\min}^{1}=\Delta_{\min}^{2}=0.10$. We focus on the priority-based regret in this section. Additional experiments that also consider the priority-free regret are given in the appendix.

In Setting 1, apart from the lexicographic optimal arm, there is another arm that is also optimal in objective 1, which requires the learner to consider rewards in objective 2. However, the third arm makes this tricky. It is not only suboptimal in objective 1 but also has very high reward in objective 2. Setting 2 is specifically designed to be challenging for Cases 1 and 2. Since arms that are not lexicographic optimal are suboptimal in exactly one objective, eliminating arms based on information from the other objective is not possible. Setting 3 contrasts with Setting 1. Unlike Setting 1 in which the expected reward of arm 3 in objective 2 is much higher than the lexicographic optimal expected reward, in Setting 3, it is much lower. However, the gap of arm 3 in objective 2 in Setting 3 is same as the absolute gap of arm 3 in Setting 1. For all cases, we set $T=10^{5}$ and average the regret of the learners over 100 individual runs. We consider OM-LEX, NOM-LEX, and PF-LEX with prior knowledge and parameters that are summarized in Table 2. For PF-LEX, we do not consider the choices for $\varepsilon$ and $\delta$ given in Theorem 3 because they require a large number of rounds for the initial exploration stage of the algorithm. Instead, we consider different exponents of $T$ as both $\varepsilon$ and $\delta$ (see the appendix for additional discussion and a result which shows the regret of PF-LEX for $\varepsilon=\delta=T^{-1/3}$ for $T=5\times 10^{8}$).

Table 8 shows the regrets of OM-LEX 1, NOM-LEX 1, 2, 3, and PF-LEX 1, 2 in Settings 1, 2 and 3 at $T=10^{5}$. There, we also report the performance of the variants of OM-LEX and NOM-LEX which only learn from the first objective and ignore the second objective, i.e., they act as if $D=1$. Note that all settings are equivalent for objective 1.

By looking at the regrets in objective 1 of OM-LEX 1, NOM-LEX 1, and their single objective variants, we observe how information from objective 2 helps learning in objective 1. OM-LEX takes advantage of large absolute gaps independent from whether the actual mean reward is higher or lower than the mean reward of arm 1. As a result, in Settings 1 and 3, it achieves lower regret in objective 1 than its single objective variant does. NOM-LEX is capable of doing this only when the gap is positive, a large absolute gap is not sufficient. As a result, only in Setting 3, it outperforms its single objective variant. In Setting 2, where information from objective 2 is not as useful as it is in Settings 1 and 3 to rule out the suboptimal arm in objective 1, OM-LEX 1 and NOM-LEX 1 perform worse than the other settings in objective 1.

By looking at the regrets of NOM-LEX 1, 2, and 3, we observe how different prior information affects the performance of NOM-LEX. Consistent with the proven regret bounds, knowing near optimal expected rewards that are closer to the lexicographic optimal ones decreases the regret in all objectives. When the near optimal expected rewards are extremely close to the lexicographic optimal ones, the performance of NOM-LEX is very similar to that of OM-LEX.

6 Conclusion and Future Work

We proposed a new multiobjective MAB problem, called the Lex-MAB, where the learner aims to minimize its regret with respect to lexicographic optimal arms. We studied the Lex-MAB with and without prior information on the expected rewards. We proved that the regret is uniformly bounded when the learner knows either the lexicographic optimal expected rewards or near-lexicographic optimal expected rewards. We also made a connection between knowing near-lexicographic optimal expected rewards and satisficing, and proved that uniformly bounded regret can be achieved for the MAB with satisficing objectives. Interestingly, we showed that the regret incurred in an objective also depends on the prior information and observations from the other objectives. We also showed that the learner can still achieve $\tilde{O}(T^{2/3})$ gap-free regret without prior information. An important open question is whether $O(\sqrt{T})$ gap-free regret can be achieved in the Lex-MAB. This will answer the question of whether learning in the Lex-MAB without prior information is fundamentally more difficult than learning in the classical stochastic MAB without prior information. The case where there is prior information only for a subset of objectives is also worth investigating in the future. Another important future research direction is to extend the current work in the multiobjective MAB to minimize the regret with respect to a target arm other than a lexicogrpahic optimal arm by exploiting the prior information about the expected rewards of the target arm.

Appendix A Appendix

A.1 Additional Theorems

Theorem 4**.**

When OM-LEX is run for Case 1, $\forall i\in\mathcal{D}$ and $\forall T\geq 1$, we have

[TABLE]

Proof.

Note that

[TABLE]

where we prove (1) by replacing $\mathcal{S}_{*}^{i}$ with $\mathcal{S}^{i}$ in the proof of Theorem 1. ∎

Theorem 5**.**

Redefine $\Delta_{\min}^{i}:=\min_{a\in\mathcal{S}^{i}}\Delta_{a}^{i}$, $\forall i\in\mathcal{D}$. When NOM-LEX is run for Case 2, $\forall i\in\mathcal{D}$ and $\forall T\geq 1$, we have

[TABLE]

Proof.

We redefine $\Delta_{\min}^{i}$ as stated in the theorem. Then,

[TABLE]

where we prove (2) by replacing $\mathcal{S}_{*}^{i}$ with $\mathcal{S}^{i}$ in the proof of Theorem 2. ∎

Note that redefining $\Delta_{\min}^{i}$ as $\min_{a\in\mathcal{S}^{i}}\Delta_{a}^{i}$ in Case 2 implies that the learner has stronger prior knowledge on the near-lexicographic optimal expected rewards, since $\min_{a\in\mathcal{S}^{i}}\Delta_{a}^{i}\leq\min_{a\in\mathcal{S}^{i}_{*}}\Delta_{a}^{i}$.

A.2 Multiobjective MAB with Satisficing Objectives

In this section, we extend the satisfaction-in-mean-reward problem introduced in [21] to the multiobjective setting. We keep the same system model but introduce the concept of satisficing optimality and a new notion of regret that captures this concept.

Satisficing Optimality: In the satisficing setting, the learner is given a target threshold $\eta_{i}$ for each objective $i\in\mathcal{D}$. We say that an arm $a$ is satisficing “optimal” or simply satisficing in objective $i$ if and only if its mean reward in objective $i$ is equal to or larger than the corresponding target threshold. Let $\mathcal{A}_{s}^{i}:=\{a\in\mathcal{A}:\mu_{a}^{i}\geq\eta_{i}\}$ be the set of satisficing arms in objective $i$ and $\mathcal{S}_{s}^{i}:=\mathcal{A}-\mathcal{A}_{s}^{i}$ be the set of non-satisficing arms in objective $i$. The satisficing goal is to play arms that are satisficing in all objectives. We assume such arms exist and call them satisficing “optimal” arms. Then, we use $*$ to denote an arbitrary satisficing “optimal” arm and call it the “optimal” satisficing arm. Note that $\eta_{i}\leq\mu_{*}^{i}$ for all $i\in\mathcal{D}$ and define $\delta_{i}:=\mu_{*}^{i}-\eta_{i}$ for all $i\in\mathcal{D}$.

Regret Definition: The satisficing regret in objective $i$ is given as $\mathrm{Reg}_{s}^{i}(T):=\sum_{t=1}^{T}(\Delta_{a(t)}^{i}-\delta_{i})\mathbb{I}\{a(t)\in\mathcal{S}_{s}^{i}\}$ and the satisficing regret is defined as the tuple $\bm{\mathrm{Reg}}_{s}(T):=(\mathrm{Reg}_{s}^{1}(T),\allowbreak\ldots,\allowbreak\mathrm{Reg}_{s}^{D}(T))$. Note that an arm $a$ incurs regret in objective $i$ only when it is not satisficing in that objective and the amount of regret incurred is equal to the gap between its mean reward in objective $i$ and the corresponding target threshold, i.e., $\Delta_{a}^{i}-\delta_{i}=\eta_{i}-\mu_{a}^{i}$.

Remark 4**.**

When $D=1$, the multiobjective formulation reduces to the exact same problem introduced in [21] as Problems 1 and 2 (satisfaction-in-mean-reward problem).

Assuming $\eta_{i}=0$ for all $i\in\mathcal{D}$ without any loss of generality,999See Remark 1 for a detailed discussion of why the generality is not lost. the algorithm proposed for Case 2, which is NOM-LEX, can also be used to solve the satisficing goal in the multiobjective MAB. Since the goal now is to minimize the satisficing regret rather than the lexicographic regret, we no longer need $\eta_{i}$ to lie between the lexicographic optimal and the second highest lexicographic optimal expected rewards in objective $i$. The following theorem bounds the expected satisficing regret for NOM-LEX.

Theorem 6**.**

When NOM-LEX is run for the satisficing goal, $\forall i\in\mathcal{D}$ and $\forall T\geq 1$, we have

[TABLE]

Proof.

Replacing lexicographic optimality with satisficing optimality, $\mathcal{S}_{*}^{i}$ with $\mathcal{S}_{s}^{i}$, and every instance of the exact phrase $\Delta_{a}^{i}$ (and $\Delta_{a(t)}^{i}$) with $\Delta_{a}^{i}-\delta_{i}$ (and $\Delta_{a(t)}^{i}-\delta_{i}$), the proof of Theorem 2 holds for Theorem 6 as well. Note that

[TABLE]

for all $i\in\mathcal{D}$. ∎

Corollary 1**.**

When NOM-LEX is run for the single objective satisfaction-in-mean-reward problem ($D=1$), $\forall T\geq 1$, we have

[TABLE]

Remark 5**.**

Theorem 6 and Corollary 1 show that bounded regret is possible for satisfaction-in-mean-reward problem whether it is single objective or multiobjective, when the learner is given the target thresholds. This result is directly in conflict with Corollary 2 of [21], which claims a logarithmic lower bound on the single objective case, and suggests that satisfaction-in-mean-reward UCL algorithm given in Section VI-A of [21] is not optimal since it fails to achieve bounded regret.

A.3 Additional Experiments

Figure 1 shows the regrets of OM-LEX 1, NOM-LEX 1, PF-LEX 1 in Setting 1. We observe that the regret of OM-LEX in objective 1 is significantly smaller than the regret of NOM-LEX. We believe this is the case because OM-LEX is able to take advantage of the large absolute gap of arm 3 to eliminate it early on, whereas NOM-LEX cannot. The behavior of PF-LEX is explained as follows. Until around round 30000, it explores all three arms uniformly since their estimated rewards in objective 1 are still chained to each other. At round 30000, the gap between the arms is deemed small enough with respect to the time horizon of the problem. In the remaining rounds, it plays only the optimistic near-lexicographic optimal arm in objective $2$ ($\hat{a}^{2}_{*}(t)$). For this case, $\varepsilon$ matches with the minimum suboptimality gap. Thus, although PF-LEX always chooses $\hat{a}^{2}_{*}(t)$, because $\hat{{\cal A}}^{1}_{*}(t)={\cal A}^{1}_{*}$, it learns to play optimally. As a remark, we note that PF-LEX could incur high regret in objective $1$ (see PF-LEX 2 in Table 8) if the minimum suboptimality gap were smaller than $\varepsilon$.

Next, for Setting 1, we consider PF-LEX 3 that has parameters $\varepsilon=\delta=T^{-1/3}$ as given in Table 4 (that match with the optimal choice for $\varepsilon$ given in Theorem 3), run simulations for $T=5\times 10^{8}$, and report the average regret of the learner over 5 runs (Figure 2). This result illustrates the identifiability problem introduced earlier that makes learning lexicographic optimal arms particularly challenging. We see that PF-LEX rules out arm 3 as a potential lexicographic optimal arm and stops incuring regret in objective 1 very early on. However, since it is not possible to be confident in that both arm 1 and arm 2 have equal expected rewards in objective 1, the algorithm still keeps exploring them uniformly until around round $2\times 10^{8}$. During this exploration stage, it incurs linear regret in objective 2. Once both arms are deemed to be optimal in objective 1, PF-LEX starts exploiting the optimistic near-lexicographic optimal arm in objective 2, after which the increase of the regret in objective 2 drops drastically.

We also present experiments in two additional settings with $D=3$. In Setting 4, there are 43 arms with expected reward vectors in $\{0.90,0.50,0.40,0.10\}^{3}$ such that each arm has a unique expected reward vector, where we eliminated arms that lexicographically dominate $(0.50,0.50,0.50)$ so that it is the only lexicographic optimal arm and $\Delta_{\min}^{1}=\Delta_{\min}^{2}=\Delta_{\min}^{3}=0.10$. Setting 4 features a large variety of arms with combinations of expected rewards that are much higher than, equal to, slightly lower than, and much lower than the lexicographic optimal expected rewards in all objectives. In Setting 5, there are 19 arms, where we eliminated arms in $\mathcal{S}_{*}^{2}$ from Setting 4 so that $\mathcal{S}_{*}^{2}=\text{\O}$ and $\Delta_{\min}^{2}=\infty$ while $\Delta_{\min}^{1}=\Delta_{\min}^{3}=0.10$. Table 4 shows OMG-LEX and NOM-LEX with different prior knowledge and parameters than the ones considered in Section 5.

We run simulations with $T=10^{5}$ and average the regret of the learners over 100 individual runs. Different from simulations in Section 5, we give results for priority-free regrets as well.

Table 5 shows the priority-based and priority-free regrets of OM-LEX 2, NOM-LEX 4 and 5 in Settings 4 and 5 at $T=10^{5}$. Since NOM-LEX considers arms with very high expected rewards compared to the near optimal expected reward as potential optimal arms, it tends to incur a lot more negative regret in priority-free settings as opposed to OM-LEX, which only looks for arms with expected rewards that are very close to the lexicographic optimal expected rewards.

In Setting 5,111111Note that the prior information of NOM-LEX 5 is not valid for Setting 4. note that any $\delta_{2}>0$ would guarantee a bounded regret for NOM-LEX (since $\Delta_{\min}^{2}=\infty$). Moreover, $\delta_{2}$ appears in none of our regret bounds in Theorems 2 and 5 for Setting 5. However, our numerical experiments show that it still affects the regret. This is because, knowing a larger $\delta_{2}$ better captures the information $\Delta_{\min}^{2}=\infty$ and results in having smaller regret in objective 1.

A.4 Proof of Theorem 1

We use the following fact to prove both Theorem 1 and 2.

Fact 1** (Results from the proof of Theorem 9 in garivier [12]).**

Given $\Delta>0$, for all arms $a\in\mathcal{A}$, for all objectives $i\in\mathcal{D}$ and for all rounds $t\in\{1,2,\ldots,T\}$, we have

[TABLE]

For an arm $a$ that is not lexicographic optimal, let $\dagger(a):=\operatorname{argmax}_{j\in\mathcal{D}}\nabla_{a}^{j}$ so that $\nabla_{a}^{\dagger(a)}=\nabla_{a}^{\max}$. When $a$ can be inferred from the context, $\dagger(a)$ is denoted by $\dagger$ only. For all objectives $i\in\mathcal{D}$, we decompose $\mathbb{E}[\mathrm{Reg}^{i}(T)]$ as

[TABLE]

Bounding (3) is trivial, since each arm is played exactly once for rounds $t\leq A$. We have

[TABLE]

In order to bound (4), we define $\tau_{a}(w)$ as the $w$th round for which $a(t)=a$, and $w_{a}(T)$ as the number of rounds for which $a(t)=a$ by round $T$. By definition, $N_{a}(\tau_{a}(w+1))=w$, $\tau_{a}(1)\leq A$ and $w_{a}(T)\leq T$ hold for all arms $a$. Thus, we have

[TABLE]

When $\mu_{a}^{\dagger}=\nabla_{a}^{\dagger}$, we have

[TABLE]

where (7) is due to Fact 1.

Similarly, when $\mu_{a}^{\dagger}=-\nabla_{a}^{\dagger}$, we have

[TABLE]

where (8) is again due to Fact 1.

Combining (7) and (8), we obtain

[TABLE]

In order to bound (5), we observe that $t>A\wedge|\hat{\mu}_{a}^{\dagger}(t)|\geq\sqrt{4\log N_{a}(t)/N_{a}(t)}\wedge a(t)=a$ can only occur during an exploration stage, where each arm is played successively. Hence, we can infer that

1. (i)

$a(t-a+*)=*$, 2. (ii)

$t-a+1>A$, 3. (iii)

$\hat{\mathcal{A}}_{*}(t-a+1)=\text{\O}$, which implies that there exists an objective $j$ such that

[TABLE]

since arm $*$ is not played after round $t-a+1$ until round $t-a+*$.

Using these observations and defining $t_{a}:=t-a+*$, we obtain

[TABLE]

where (9) is due to Hoeffding’s inequality for sub-Gaussian random variables [8].

A.5 Proof of Theorem 2

For an arm $a$ that is not lexicographic optimal, let $\dagger(a):=\operatorname{argmax}_{j\in\mathcal{D}}\Delta_{a}^{j}-\delta_{j}$. When $a$ can be inferred from the context, $\dagger(a)$ is denoted by $\dagger$ only. For all objectives $i\in\mathcal{D}$, we decompose $\mathbb{E}[\mathrm{Reg}^{i}(T)]$ as

[TABLE]

Bounding (10) is trivial, since each arm is played exactly once for rounds $t\leq A$. We have

[TABLE]

In order to bound (11), we use $\tau_{a}(w)$ and $w_{a}(T)$ defined in the proof of Theorem 1. We have

[TABLE]

where (13) is due to Fact 1.

In order to bound (12), we observe that $t>A\wedge\hat{\mu}_{a}^{\dagger}(t)\leq-\sqrt{4\log N_{a}(t)/N_{a}(t)}\wedge a(t)=a$ can only occur during an exploration stage, where each arm is played successively. Hence we can infer that

1. (i)

$a(t-a+*)=*$, 2. (ii)

$t-a+1>A$, 3. (iii)

$\hat{\mathcal{A}}_{*}(t-a+1)=\text{\O}$, which implies that there exists an objective $j$ such that

[TABLE]

since arm $*$ is not played after round $t-a+1$ until round $t-a+*$.

Using these observations and defining $t_{a}:=t-a+*$, we obtain

[TABLE]

where (14) holds since $\mu_{*}^{j}=\delta_{j}\geq 0$ and (15) is due to Hoeffding’s inequality for sub-Gaussian random variables [8].

A.6 Proof of Theorem 3

First, we state a concentration inequality that will be used in the proof.

Lemma 1**.**

(Lemma 6 in abbasi2011improved [23]) Consider an arm $a$ for which the rewards of objective $i$ are generated by a process $\{R^{i}_{a}(t)\}_{t=1}^{T}$ with $\mu^{i}_{a}=\text{E}[R^{i}_{a}(t)]$, where the noise $R^{i}_{a}(t)-\mu^{i}_{a}$ is conditionally 1-sub-Gaussian. Let $N_{a}(T)$ denote the number of times $a$ is selected by the beginning of round $T$. Let $\hat{\mu}_{a}(T)=\sum_{t=1}^{T-1}\text{I}(a(t)=a)R^{i}_{a}(t)/N_{a}(T)$ for $N_{a}(T)>0$ and $\hat{\mu}_{a}(T)=0$ for $N_{a}(T)=0$. Then, for any $0<\delta<AD$ with probability at least $1-\delta/(AD)$ we have

[TABLE]

Let $\text{UC}^{i}_{a}:=\cup_{t=1}^{T}\{\mu^{i}_{a}\notin[l^{i}_{a}(t),u^{i}_{a}(t)]\}$, $\text{UC}^{i}:=\cup_{a\in{\cal A}}\text{UC}^{i}_{a}$ and $\text{UC}:=\cup_{i\in{\cal D}}\text{UC}^{i}$. The following lemma bounds the probability of UC.

Lemma 2**.**

$\mathbb{P}(\text{UC})\leq\delta$.

Proof.

This follows from Lemma 1. We observe that $\{\mu^{i}_{a}\in[l^{i}_{a}(t),u^{i}_{a}(t)]\}=\{|\mu^{i}_{a}-\hat{\mu}^{i}_{a}(t)|\leq c_{a}(t)\}$. Thus, Lemma 1 shows that $\neg\text{UC}^{i}_{a}$ holds with probability at least $1-\delta/(AD)$, and hence, $\text{UC}^{i}_{a}$ holds with probability at most $\delta/(AD)$. Applying the union bound, we get $\mathbb{P}(\text{UC})\leq\delta$. ∎

Let ${\cal T}:=\{1\leq t\leq T:\forall a\in\hat{\mathcal{A}}_{*}^{1}(t):c_{a}(t)\leq\varepsilon/2\}$ denote the set of rounds in which PF-LEX selects the arm $\hat{a}_{*}^{D}(t)$ and $\neg{\cal T}:=\{1,\ldots,T\}-{\cal T}$. In the following lemma, the gap of the arm selected in round $t\in{\cal T}$ in objective $i$ is bounded as a function of $\varepsilon$ and the length of the confidence interval of the selected arm on event $\neg\text{UC}$ if the selected arm is in $\mathcal{S}_{*}^{i}$.

Lemma 3**.**

When PF-LEX is run, the following holds on event $\neg\text{UC}$ if $a(t)\in\mathcal{S}_{*}^{i}$: $\mu_{*}^{i}-\mu_{a(t)}^{i}\leq u_{a(t)}^{i}(t)-l_{a(t)}^{i}(t)+\varepsilon(A-1)$ for $t\in{\cal T}$.

Proof.

Consider any lexicographic optimal arm $*$. We have

[TABLE]

Here (16) holds since $\mu_{*}^{i}\leq u_{*}^{i}(t)$ and $\mu_{a(t)}^{i}\geq l_{a(t)}^{i}(t)$ on event $\neg\text{UC}$. (17) holds by the definition of $\hat{a}_{*}^{i}(t)$ and the fact that $*\in\hat{\mathcal{A}}_{*}^{i-1}(t)$, which is proven by induction. For this, consider any objective $j\in\{1,\ldots,i-1\}$. We first observe that $*\in\hat{\mathcal{A}}_{*}^{0}(t)$ and $a(t)=\hat{a}_{*}^{D}(t)\in\hat{\mathcal{A}}_{*}^{D-1}(t)\subseteq\hat{\mathcal{A}}_{*}^{j}(t)$. Next, we show that $*\in\hat{\mathcal{A}}_{*}^{j-1}(t)\implies*\in\hat{\mathcal{A}}_{*}^{j}(t)$ to conclude that $*\in\hat{\mathcal{A}}_{*}^{i-1}(t)$. Since $a(t)\in\mathcal{S}_{*}^{i}$, $\mu_{a(t)}^{j}=\mu_{*}^{j}$, which implies that $a(t)$ and $*$ are linked in objective $j$. Since $a(t)\in\hat{\mathcal{A}}_{*}^{j}(t)$, $a(t)$ is chained to $\hat{a}_{*}^{j}(t)$ in objective $j$, which implies that $*$ is chained to $\hat{a}_{*}^{j}(t)$ in objective $j$ as well. Finally, if $i=D$, (18) holds trivially as $a(t)=\hat{a}_{*}^{i}(t)=\hat{a}_{*}^{D}(t)$. Otherwise, since $a(t)=\hat{a}_{*}^{D}(t)\in\hat{\mathcal{A}}_{*}^{D-1}(t)\subseteq\hat{\mathcal{A}}_{*}^{i}(t)$, $a(t)$ is chained to $\hat{a}_{*}^{i}(t)$, which implies that $|u_{\hat{a}_{*}^{i}(t)}^{i}(t)-u_{a(t)}^{i}(t)|\leq 2(|\hat{\mathcal{A}}_{*}^{i}(t)|-1)\max_{a\in\hat{\mathcal{A}}_{*}^{i}(t)}c_{a}(t)\leq\varepsilon(A-1)$. ∎

We also need to bound the regret in objective $i$ for rounds up to round $T$ for which $t\notin{\cal T}$. Let $\neg{\cal T}_{a}:=\{t\in\neg\mathcal{T}:a(t)=a\}$. Obviously, PF-LEX does not incur any regret in objective $i$ in rounds $t\in\neg{\cal T}_{a}$ for $a\in{\cal A}-{\cal S}^{i}_{*}$, and incurs regret $\Delta^{i}_{a}$ in objective $i$ in rounds $t\in\neg{\cal T}_{a}$ for $a\in{\cal S}^{i}_{*}$.

Lemma 4**.**

When PF-LEX is run, we have

[TABLE]

for all objectives $i\in\mathcal{D}$.

Proof.

The proof follows from bounding the cardinality of $\neg{\cal T}_{a}$ for $a\in{\cal S}^{i}_{*}$. Note that $t\in\neg{\cal T}_{a}$ happens only when $c_{a}(t)>\varepsilon/2$. Similar to the proof of Theorem 7 in abbasi2011improved [23], this implies that

[TABLE]

Then, from Lemma 8 in antos2010 [24], we obtain $N_{a}(t)\leq 3+\frac{16}{\varepsilon^{2}}\log\frac{2\sqrt{e}AD}{\varepsilon\delta}$. ∎

In the remaining part, we bound $\mathrm{Reg}_{pb}^{i}(T)$ under the event $\neg\text{UC}$ and $\mathbb{E}[\mathrm{Reg}_{pb}^{i}(T)]$ by using the results of the lemmas above. For the latter, we observe that:

[TABLE]

For each $i\in\mathcal{D}$, the bound for $\mathrm{Reg}_{pb}^{i}(T)$ is obtained by using the result in Lemmas 3 and 4. By Lemma 4, we know that

[TABLE]

Let ${\cal N}_{a}:=\{t\in\mathcal{T}:a(t)=a\}$. By Lemma 3, on event $\neg\text{UC}$ (which happens with probability at least $1-\delta$), we have

[TABLE]

The bound for $\mathrm{Reg}_{pb}^{i}(T)$ is obtained by summing the results of (20) and (21). Finally, the bounds on the expected regret simply follows from using (19) and setting $\delta=1/T$.

A.7 Tables of Notation

General notation is listed in Table 6. Notations specific to each case covered in Section 4 are listed in Tables 7, 8 and 9 respectively.