Risk-Averse Explore-Then-Commit Algorithms for Finite-Time Bandits

TL;DR

This paper introduces risk-averse explore-then-commit algorithms for finite-time bandit problems, focusing on selecting arms with the best risk-return trade-off rather than just the highest expected reward, with proven regret bounds and robust performance.

Contribution

It proposes two novel risk-averse explore-then-commit algorithms that do not rely on hyper-parameters and provide finite-time regret guarantees, improving robustness over existing methods.

Findings

Algorithms outperform existing risk-averse bandit algorithms in robustness.

Finite-time regret bounds are established for the proposed methods.

Numerical evaluations confirm practical effectiveness.

Abstract

In this paper, we study multi-armed bandit problems in explore-then-commit setting. In our proposed explore-then-commit setting, the goal is to identify the best arm after a pure experimentation (exploration) phase and exploit it once or for a given finite number of times. We identify that although the arm with the highest expected reward is the most desirable objective for infinite exploitations, it is not necessarily the one that is most probable to have the highest reward in a single or finite-time exploitations. Alternatively, we advocate the idea of risk-aversion where the objective is to compete against the arm with the best risk-return trade-off. Then, we propose two algorithms whose objectives are to select the arm that is most probable to reward the most. Using a new notion of finite-time exploitation regret, we find an upper bound for the minimum number of experiments before commitment, to guarantee an upper bound for the regret. As compared to existing risk-averse bandit algorithms, our algorithms do not rely on hyper-parameters, resulting in a more robust behavior in practice, which is verified by the numerical evaluation.

Equations

$$k^{\ast }=\underset{k}{\mathrm{argmax}}P(R_k^M\ge {\mathbf{R}}_{-k}^M),$$

$$r_M(\Delta p)=P\left(p_{k^{\ast }}^M-p_{\hat{k}}^M\ge \Delta p\right).$$

$$f_1(u)=\alpha e^{-2(u-3{)}^2}\cdot \text{mathds}1\{0\le u\le 10\}$$

$$f_2(u)=\beta \left(3e^{-8(u-1{)}^2}+2e^{-8(u-8{)}^2}\right)\cdot \text{mathds}1\{0\le u\le 10\},$$

$$k^{\ast }=\underset{k}{\mathrm{argmax}}P(R_k\ge {\mathbf{R}}_{-k}),$$

$${\hat{p}}_k=\frac{{\sum }_{n_1=1}^N{\sum }_{n_2=1}^N\cdots {\sum }_{n_K=1}^N\text{mathds}1\{r_{k,n_k}\ge {\mathbf{r}}_{-k,n_{-k}}\}}{N^K},$$

$${\hat{p}}_k=\frac{{\sum }_{n=1}^N\text{mathds}1\{r_{k,n}\ge {\mathbf{r}}_{-k,n}\}}{N}.$$

$$r(\Delta p)=P\left(p_{k^{\ast }}-p_{\hat{k}}\ge \Delta p\right).$$

$$P\left(p_k\in \left({\hat{p}}_k-\frac{a}{2\sqrt{N}},{\hat{p}}_k+\frac{a}{2\sqrt{N}}\right)\right)\ge 1-2e^{-\frac{a^2}{2}},\forall k\in \mathcal{K}.$$

$$\left\{p_{k^{\ast }}-p_{\hat{k}}\ge \Delta p\right\}\subseteq$$

$$\left\{\exists k\in \mathcal{K}\text{ such that }{p}_k\notin \left({\hat{p}}_k-\frac{\Delta p}{2},{\hat{p}}_k+\frac{\Delta p}{2}\right)\right\}\stackrel{(a)}{\subseteq }$$

$$\left\{\exists k\in \mathcal{K}\text{ such that }{p}_k\notin \left({\hat{p}}_k-\frac{a}{2\sqrt{N}},{\hat{p}}_k+\frac{a}{2\sqrt{N}}\right)\right\},$$

$$r(\Delta p)=P\left(p_{k^{\ast }}-p_{\hat{k}}\ge \Delta p\right)\le 2K{e}^{-\frac{a^2}{2}}={ϵ}_r.$$

$$r=\underset{\Delta p>0}{\inf }r(\Delta p)=P\left(\hat{k}\ne k^{\ast }\right),$$

$${\mathcal{R}}_k^M=\left\{\sum _{n\in S_{\mathcal{K}}}{r}_{k,n}\text{ s.t. }{S}_{\mathcal{K}}\subseteq \{1,2,\dots ,N\}\text{ and }|S_{\mathcal{K}}|=M\right\}.$$

$${\hat{p}}_k^M=\frac{{\sum }_{j_1=1}^{\left(\genfrac{}{}{0ex}{}{N}{M}\right)}{\sum }_{j_2=1}^{\left(\genfrac{}{}{0ex}{}{N}{M}\right)}\cdots {\sum }_{j_K=1}^{\left(\genfrac{}{}{0ex}{}{N}{M}\right)}\text{mathds}1\{r_{k,j_k}^M\ge {\mathbf{r}}_{-k,j_{-k}}^M\}}{{\left(\genfrac{}{}{0ex}{}{N}{M}\right)}^K}.$$

$${\hat{p}}_k^M=\frac{{\sum }_{j=1}^{\left(\genfrac{}{}{0ex}{}{N}{M}\right)}\text{mathds}1\{r_{k,j}^M\ge {\mathbf{r}}_{-k,j}^M\}}{\left(\genfrac{}{}{0ex}{}{N}{M}\right)}.$$

$$r_M=\underset{\Delta p>0}{\inf }{r}_M(\Delta p)=P\left(\hat{k}\ne k^{\ast }\right),$$

$$f_1(u)=\alpha e^{-0.5(u-2{)}^2}\cdot \text{mathds}1\{0\le u\le 10\}$$

$$f_2(u)=\beta e^{-0.5(u-1{)}^2}\cdot \text{mathds}1\{0\le u\le 10\},$$

$$r^{\ast }=$$

$$\frac{1}{2}\cdot \left(\genfrac{}{}{0ex}{}{N}{\frac{N}{2}}\right)\cdot (1-p_{k^{\ast }}{)}^{\frac{N}{2}}\cdot p_{k^{\ast }}^{\frac{N}{2}}\cdot \text{mathds}1\{\text{N is even}\}.$$

$$\underset{N}{\mathrm{argmin}}C(N)+\alpha \cdot {\bar{r}}^{\ast },$$

$$P\left(p_k^M\in \left({\hat{p}}_k^M-\frac{a}{2\sqrt{\lfloor \frac{N}{M}\rfloor }},{\hat{p}}_k^M+\frac{a}{2\sqrt{\lfloor \frac{N}{M}\rfloor }}\right)\right)\ge 1-2e^{-\frac{a^2}{2}},$$

$$\left\{p_{k^{\ast }}^M-p_{\hat{k}}^M\ge \Delta p\right\}\subseteq$$

$$\left\{\exists k\in \mathcal{K}\text{ s.t. }{p}_k^M\notin \left({\hat{p}}_k^M-\frac{\Delta p}{2},{\hat{p}}_k^M+\frac{\Delta p}{2}\right)\right\}\stackrel{(a)}{\subseteq }$$

$$\left\{\exists k\in \mathcal{K}\text{ s.t. }{p}_k^M\notin \left({\hat{p}}_k^M-\frac{a}{2\sqrt{\lfloor \frac{N}{M}\rfloor }},{\hat{p}}_k^M+\frac{a}{2\sqrt{\lfloor \frac{N}{M}\rfloor }}\right)\right\},$$

$$r_M(\Delta p)=P\left(p_{k^{\ast }}^M-p_{\hat{k}}^M\ge \Delta p\right)\le 2K{e}^{-\frac{a^2}{2}}={ϵ}_r.$$

Full text

I Introduction

One of the classes of decision making models is the multi-armed bandit (MAB) framework where decision makers learn the model of different arms that are unknown and actions do not change the state of arms [1]. The MAB problem was originally proposed by Robbins [2], and has a wide range of applications in finance [3, 4], health-care [5], autonomous vehicles [6, 7], communication and networks [8, 9, 10, 11, 12], and energy management [13, 14] to name but a few. In the classical MAB problem, the decision-maker sequentially selects an arm (action) with an unknown reward distribution out of $K$ independent arms. The random reward of the selected arm is revealed and the rewards of other arms remain unknown. At each step, the decision-maker encounters a dilemma between exploitation of the best identified arm versus exploration of alternative arms. The goal of the classical model of multi-armed bandit is to maximize the expected cumulative reward over a time horizon.

In this paper, we focus on a setting where a player is allowed to explore different arms in the exploration (or experimentation, used interchangeably) phase before committing to the best identified arm for exploitation in one or a given finite number of times. This setting of interest is motivated by several application domains such as personalized health-care and one-time investment. In such applications, exploitation is costly and/or it is infeasible to exploit for a large number of times, but arms can be experimented by simulation and/or based on the historical data for multiple times with negligible cost [15]. The big step in personalized health-care is to provide an individual patient with his/her disease risk profile based on his/her electronic medical record and personalized assessments [16, 17]. The different treatments (arms) are evaluated for a person by simulation or mice trials for many times with a low cost, but one personalized treatment is exploited once for a patient in the end [18, 19]. Another example of one-time exploitation is one-time investment where an investor chooses a factory out of multiple ones. Based on experimentation on historical data, he/she selects a factory to invest in once. The common theme in both above examples is to identify the best arm for one-time exploitation after an experimentation phase of pure exploration.

The above setting falls in the class of MAB problems called explore-then-commit. The previous works [15, 20, 21, 22, 13, 23] on explore-then-commit bandits, to the best of our knowledge, try to identify the arm with an optimum risk-return criterion on an expectation sense up to a hyper-parameter. Even though this objective is desirable in the settings with infinite exploitations, it is not necessarily the best objective in the explore-then-commit setting with a single or finite exploitations. We further elaborate on this observation by an illustrative example in Section III. We advocate an alternative approach in which the objective is to select an arm that is most probable to reward the most. It has been realized that in many scenarios of multi-armed bandits, considering maximum expected reward as an objective to select an arm is not the best strategy. In such scenarios, players not only aim to achieve the maximum cumulative reward, but they also want to minimize the uncertainty such as risk in the outcome [24], and scuh approaches are known as risk-averse MAB. In literature, there are several approaches to address the risk-averse MAB including mean-variance (MV) [23] and the conditional value at risk (CVaR) [13]. The performance of both MV and CVaR, are highly dependent on different single scalar hyper-parameters, and selecting an inappropriate hyper-parameter might degrade the performance substantially. More details on MV and CVaR criteria are given in Section II, and the negative impact of hyper-parameter mismatch is studied in Section V.

Contributions: We propose a class of hyper-parameter-free risk-averse algorithms (called OTE/FTE-MAB) for explore-then-commit bandits with finite-time exploitations. The goal of the algorithms is to select the arm that is most probable to give the player the highest reward. To analyze the algorithms, we define a new notion of finite-time exploitation regret for our setting of interest. We provide concrete mathematical support to obtain an upper bound of order $\ln\left(\frac{1}{\epsilon}\right)$ for the minimum number of experiments that should be done to guarantee upper bound $\epsilon$ for regret. More specifically, our results show that by utilizing the proposed algorithms, the regret can be bounded arbitrarily small by sufficient number of experimentations. As a salient feature, the OTE/FTE-MAB algorithm is hyper-parameter-free, so it is not prone to errors due to hyper-parameter mismatch.

Organization of the Paper: Section II discusses related work. In Section III, the one/finite-time exploitation multi-armed bandit problem after an experimentation phase is formally described. We define a new notion of one/finite-time exploitation regret for our problem setup. An example is provided clarifying the motivation of our work. In Section IV, we propose the OTE-MAB and FTE-MAB algorithms, and find an upper bound of order $\ln\left(\frac{1}{\epsilon}\right)$ for the minimum number of pure explorations needed to guarantee upper bound $\epsilon$ for regret. In Section V, we evaluate the OTE-MAB algorithm versus risk-averse baselines and compare the minimum number of experiments needed to guarantee an upper bound on regret for both the OTE-MAB and FTE-MAB algorithms. We conclude the paper with a discussion of opportunities for future work in Section VI.

II Related Work

Explore-then-commit bandit is a class of multi-armed bandit problems that has two consecutive phases named as exploration (experimentation) and commitment. The decision maker can arbitrarily explore each arm in the experimentation phase; however, he/she needs to commit to one selected arm in the commitment phase. There are several studies on explore-then-commit bandits in the literature as follows. Bui et al. [15] studied the optimal number of explorations when cost is incurred in both phases. Liau et al. [25] designed an explore-then-commit algorithm for the case where there is a limited space to record the arm reward statistics. Perchet et al. [26] studied explore-then-commit policy under the assumption that the employed policy must split explorations into a number of batches. None of these works have addressed the risk-averse issue on explore-then-commit bandits. In the following, we present an overview on risk-averse bandits.

There are several criteria to measure and to model risk in a risk-averse multi-armed bandit problem. One of the common risk measurements is the mean-variance paradigm [27]. The two algorithms MV-LCB and ExpExp proposed by Sani et al. [23] are based on mean-variance concept. They define the mean-variance of an arm with mean $\mu$ and variance $\sigma^{2}$ as MV$=\sigma^{2}-\rho\cdot\mu$, where $\rho\geq 0$ is the absolute risk tolerance coefficient. In an infinite horizon multi-armed bandit problem, MV-LCB plays the arm with minimum lower confidence bound for estimation of MV. In a best-arm identification setting, the ExpExp algorithm explores each of the arms for the same number of times and selects the arm with minimum estimated MV. This approach is followed by numerous researchers in risk-averse multi-armed bandit problems [24, 28, 29, 30].

Another way of considering risk in multi-armed bandit problems is to use conditional value at risk level $\alpha$, CVaRα, where it is the expected policy return in a specified quantile. CVaRα is utilized by Galichet et al. [13] in risk-aware multi-armed bandit problems. They presented the Multi-Armed Risk-Aware Bandit (MaRaB) algorithm aiming to select the arm with the maximum conditional value at risk level $\alpha$, CVaRα. Formally, let $0<\alpha<1$ be the target quantile level and $v_{\alpha}$ defined as $P(R<v_{\alpha})=\alpha$ be the associated quantile value, where $R$ is the arm reward. The conditional value at risk $\alpha$ is then defined as CVaR${}_{\alpha}=\mathds{E}\left[R|R<v_{\alpha}\right]$. CVaRα is also followed by researchers in multi-armed bandit problems [24, 31, 32, 33, 34].

III Problem Statement

Consider arms $\mathcal{K}=\{1,2,\dots,K\}$ whose rewards are random variables $R_{1},R_{2},\dots,R_{K}$ that have unknown distributions $f_{1},f_{2},\dots,f_{K}$ with unknown finite expected values $\mu_{1},\mu_{2},\dots,\mu_{K}$, respectively. The goal is to identify the best arm at the end of an experimentation phase that is followed by an exploitation phase, where the best arm is exploited for a given number of times, $M<\infty$. In the experimentation phase, each arm is sampled for $N$ independent times. Denote the observed reward of arm $k\in\mathcal{K}$ at iteration $n\in\{1,2,\dots,N\}$ of experimentation by $r_{k,n}$.

Let $R_{k}^{M}=X_{1}+X_{2}+\dots+X_{M},$ where $X_{1},X_{2},\dots,X_{M}$ are independent and identically distributed random variables and $X_{1}\sim f_{k}$. The optimum arm for $M$ exploitations in the sense that maximizes the probability of receiving the highest reward is

[TABLE]

where $\boldsymbol{R}_{-k}^{M}=\{R_{1}^{M},R_{2}^{M},\dots,R_{k-1}^{M},R_{k+1}^{M},\dots,R_{K}^{M}\}$ and what $R_{k}^{M}$ being greater than or equal to a vector means is that it is greater than or equal to all elements of the vector. Let $p_{k}^{M}=P(R_{k}^{M}\geq\boldsymbol{R}_{-k}^{M})$. Given the above preliminaries, the finite-time exploitation regret is defined below.

Definition 1

The finite-time exploitation regret, $r_{M}(\Delta p)$, is defined as a function of an input $0<\Delta p<1$ for the selected arm $\hat{k}$ as

[TABLE]

Note that the above definition of regret is different from the commonly used regret in bandit problems. In the following, an example is presented that motivates to define this new notion of regret for the finite-time exploitation setting.

III-A Illustrative Example

As mentioned in the Introduction, although the arm with the highest expected reward is the optimum arm for utilization in infinite number of exploitations, it is not necessarily the one that is most probable to have the highest reward in a single or some finite number of exploitations. In the following example, two arms are considered such that $\mu_{2}>\mu_{1}$, but it is more probable that a one-time exploitation of the first arm rewards us more than a one-time exploitation of the second arm. Hence, arm $\underset{k}{\operatorname*{argmax}}\ \mu_{k}$ is not necessarily the ideal arm for one-time exploitation let alone the arm with the maximum empirical mean, i.e. $\underset{k}{\operatorname*{argmax}}\ \frac{\sum_{n=1}^{N}r_{k,n}}{n}$.

Example 1

Consider two arms with the following independent reward distributions:

[TABLE]

where $\alpha$ and $\beta$ are constants for which each of the two distributions integrate to one and $\mathds{1}\{.\}$ is the indicator function.

In example 1, although the second arm has a larger mean than the first one, $\mu_{2}\approx 3.8$ and $\mu_{1}\approx 3$, the variance of reward received from the second arm is larger than that from the first one, which increases the risk of choosing the second arm for a one-time exploitation application. In fact, the first arm with lower mean is more probable to reward us more than the second arm since $P(R_{1}\geq R_{2})\approx 0.6>0.5$. In general, a larger variance for the received reward is against the principle of risk-aversion where the objective is to keep a balance in a trade-off between the expected return and risk of an action [23]. Mean-variance is an existing approach to tackle this scenario. However, it has some drawbacks that are explained in details in the following.

The mean-variance (MV) of an arm depends on the hyper-parameter $\rho\geq 0$, which is the absolute risk tolerance coefficient. The trade-off on $\rho$ is that if it is set to zero, the arm with the minimum variance is selected. On the other hand, if $\rho$ goes to infinity, the arm with the maximum expected reward is selected, which is the same as classical multi-armed bandit approach. Although the behavior of mean-variance trade-off is known for marginal values of $\rho$, it is not obvious what value of the hyper-parameter $\rho$ keeps a desirable balance between return and risk. The choice of this hyper-parameter can be tricky and as will be shown in Section V; a bad choice can increase the regret dramatically. As a simple example, consider two arms with unknown parameters $\mu_{1}=10,\sigma_{1}^{2}=10,\mu_{2}=1,\sigma_{2}^{2}=1$, and $P(R_{1}>R_{2})=1$. The mean-variance trade-off is formalized as $\hat{\sigma}_{k}^{2}-\rho\hat{\mu}_{k}$, where $\hat{\sigma}_{k}^{2}$ and $\hat{\mu}_{k}$ are empirical estimates of variance and mean of each arm. Note that the empirical means and variances converge to true values, so the second arm that is performing worse with probability one is selected if $\rho<1$. In order to address this issue, we alternatively propose the following best arm identification algorithm for One-Time (Finite-time) Exploitation in a Multi-Armed Bandit problem (OTE/FTE-MAB algorithm) that has concrete mathematical support for its action and is hyper-parameter-free.

IV One/Finite-Time Exploitation in Multi-Armed Bandit Problems after an Experimentation Phase

In this section, we propose the OTE-MAB and FTE-MAB algorithms. The OTE-MAB algorithm is a specific case of FTE-MAB algorithm. Since the proof of theorem related to the FTE-MAB algorithm is notationally heavy, we first propose the OTE-MAB algorithm in Subsection IV-A and postpone the FTE-MAB algorithm to Subsection IV-B.

IV-A The OTE-MAB Algorithm

The OTE-MAB algorithm desires to play the arm that is most probable to reward the most for the case $M=1$ as

[TABLE]

which is a specific case of Equation (1). Due to simplicity of notation, the $M$-notation is eliminated in this subsection.

Remark 1

A more general version of the OTE-MAB algorithm is to concatenate a constant $c$ to vector $\boldsymbol{R}_{-k}$ as $\boldsymbol{R}_{-k}=\{R_{1},R_{2},\dots,R_{k-1},R_{k+1},\dots,R_{K},c\}$.

Since the reward distributions of the $K$ independent arms are not known, the exact values of $p_{k}=P(R_{k}\geq\boldsymbol{R}_{-k})$ are unknown. Hence, estimates of these probabilities, $\hat{p}_{k}$, are needed to be evaluated based on observations in the experimentation phase as follows:

[TABLE]

where \boldsymbol{r}_{-k,n_{-k}}=\big{(}r_{1,n_{1}},r_{2,n_{2}},\dots,r_{k-1,n_{k-1}},r_{k+1,n_{k+1}}, \dots,r_{K,n_{K}}\big{)} and rewards of different arms are assumed to be independent.

Remark 2

If rewards of different arms are dependent, instantaneous observations of all arms at the same time are needed for $N$ times and $\hat{p}_{k}$ is calculated as follows:

[TABLE]

The OTE-MAB algorithm selects arm $\hat{k}=\underset{k}{\operatorname*{argmax}}\ \hat{p}_{k}$ as the best arm in terms of rewarding the most with the highest probability in one-time exploitation. The one-time exploitation regret, $r(\Delta p)$, which is a specific case of Definition 2, is defined as follows, where $k^{*}$ is defined in Equation (3):

[TABLE]

The OTE-MAB algorithm is summarized in Algorithm 1. We next present a theorem on an upper bound of the minimum number of experiments needed to guarantee an upper bound on regret of Algorithm 1.

Theorem 1

For any $0<\epsilon_{r},\Delta p<1,$ if each of the $K$ arms are experimented for $N\geq\frac{2\ln\left(\frac{2K}{\epsilon_{r}}\right)}{{\Delta p}^{2}}$ times in the experimentation phase, the one-time exploitation regret defined in Equation (6) is bounded by $\epsilon_{r}$, i.e. $r(\Delta p)\leq\epsilon_{r}$. Note that simultaneous exploration of the $K$ arms are required in the experimentation phase if arm rewards are dependent.

Proof:

Consider the Bernoulli random variables $B_{k}=\mathds{1}\{R_{k}\geq\boldsymbol{R}_{-k}\}$ and their unknown means $p_{k}=\mathds{E}[B_{k}]=P(R_{k}\geq\boldsymbol{R}_{-k})$ for $k\in\mathcal{K}$. Possessing $N$ independent observations from each of the $K$ independent or dependent arms in the pure exploration phase, the confidence interval derived from Hoeffding’s inequality for estimating $p_{k}$ based on Equation (4) or Equation (5) with confidence level $1-2e^{-\frac{a^{2}}{2}}$ has the property that

[TABLE]

Note that for the case of dependent arms, there is an $N$-tuple containing the instantaneous observation of the $K$ arm rewards as $\left(r_{1,n},r_{2,n},\dots,r_{K,n}\right)$ for $n\in\{1,2,\dots,N\}$, which is used for estimation of $\hat{p}_{k}$ in Equation (5). On the other hand, for the case of independent arms, any of the $N^{K}$ orderings of the $N$ observations of the $K$ arm rewards can be used for estimation of $\hat{p}_{k}$ as is done in Equation (4). However, $\left(\hat{p}_{k}-\frac{a}{2\sqrt{N^{K}}},\hat{p}_{k}+\frac{a}{2\sqrt{N^{K}}}\right)$ cannot be used as confidence interval with confidence level $1-2e^{-\frac{a^{2}}{2}}$. The reason is that, although $\hat{p}_{k}$ is derived from $N^{K}$ samples, not all those samples are independent, but exactly $N$ of the $N^{K}$ samples are independent. In fact, the observed independent rewards can be classified as $N$-tuples of the $K$ arm rewards with independent elements in $N^{k-1}\times(N-1)^{k-1}\times\cdots\times 1^{k-1}=(N!)^{K-1}$ different ways. None of such $N$-tuples has any priority over the other ones to estimate $p_{k}$, so $\hat{p}_{k}$ can be computed based on any of the $N$-tuples. The estimate of $p_{k}$ derived from any of those $N$-tuples is in $\left(p_{k}-\frac{a}{2\sqrt{N}},p_{k}+\frac{a}{2\sqrt{N}}\right)$ with probability at least $1-2e^{-\frac{a^{2}}{2}}$, so the average of those estimations is again in the mentioned interval with probability at least $1-2e^{-\frac{a^{2}}{2}}$. Note that the average of estimates of $p_{k}$ derived from all of the $(N!)^{K-1}$ different $N$-tuples is equal to $\hat{p}_{k}$ derived from Equation (4) due to the following reason. An element of an $N$-tuple is repeated for $((N-1)!)^{K-1}$ times in all $N$-tuples. Hence, averaging over the $\frac{(N!)^{K-1}\cdot N}{((N-1)!)^{K-1}}=N^{K}$ number of distinct elements of $N$-tuples results in the same answer as the case of averaging the estimates of $p_{k}$ derived from all of $(N!)^{K-1}$ different $N$-tuples. As a result, $\frac{a}{2\sqrt{N}}$ can be used as the half width of the confidence interval for estimators obtained from Equations (4) and (5) for both independent and dependent arms.

In order to find a bound on regret, defined in Equation (6) as $r(\Delta p)=P\left(p_{k^{*}}-p_{\hat{k}}\geq\Delta p\right)$, note that

[TABLE]

[TABLE]

where $(a)$ is true if $\frac{a}{2\sqrt{N}}\leq\frac{\Delta p}{2}$. By using union bound and Equation (17), the probability of the right-hand side of the above equation can be bounded as follows, which results in the following bound on regret:

[TABLE]

The above upper bound on regret is derived under the condition that $\frac{a}{2\sqrt{N}}\leq\frac{\Delta p}{2}$, which by using $a^{2}=2\ln\left(\frac{2K}{\epsilon_{r}}\right)$ and simple algebraic calculations is equivalent to $N\geq\frac{2\ln\left(\frac{2K}{\epsilon_{r}}\right)}{{\Delta p}^{2}}$. ∎

According to Theorem 1, the selected arm by Algorithm 1, $\hat{k}$, satisfies $p_{\hat{k}}=P\left(R_{\hat{k}}\geq\boldsymbol{R}_{-\hat{k}}\right)\geq(p_{k^{*}}-\Delta p)$ with probability at least $1-\epsilon_{r}$ for any $0<\epsilon_{r},\Delta p<1$, if each of the $K$ arms is explored in the experimentation phase for $N\geq\frac{2\ln\left(\frac{2K}{\epsilon_{r}}\right)}{{\Delta p}^{2}}$ times. Hence, $p_{\hat{k}}$ can get arbitrarily close to $p_{k^{*}}$ by increasing the number of pure explorations in the experimentation phase.

Let $p_{(1)},p_{(2)},\dots,p_{(K)}$ be the ordered list of $p_{1},$ $p_{2},$ $\dots,$ $p_{K}$ in descending order. Note that arm $(1)$ is actually arm $k^{*}$ defined in Equation (3). Define the difference between the two maximum $p_{k}$’s as $\Delta p^{*}=p_{(1)}-p_{(2)}$, where without loss of generality is assumed to be nonzero. Having the knowledge of $\Delta p^{*}$ or a lower bound on it, a stronger notion of regret can be defined as

[TABLE]

and have the following corollary.

Corollary 1

From the theoretical point of view, upon the knowledge of $\Delta p^{*}$ or a lower bound on it, for any $0<\epsilon_{r}<1$, the regret defined in Equation (10) is bounded by $\epsilon_{r},$ i.e. $r<\epsilon_{r}$, if the $K$ arms are explored for $N\geq\frac{2\ln\left(\frac{2K}{\epsilon_{r}}\right)}{{\Delta p^{*}}^{2}}$ times each. If arms are dependent, instantaneous explorations of the $K$ arms are needed.

IV-B The FTE-MAB Algorithm

Consider the case where an arm is going to be exploited for finite number of times, $M<\infty$. The best arm for $M$-time exploitations is defined in Equation (1). Since reward distributions are unknown, $p_{k}^{M}$’s are needed to be estimated based on observations in pure exploration phase. In the case of independent arms, define the vector $\mathcal{R}_{k}^{M}$ with cardinality $N\choose M$ as

[TABLE]

Let $r_{k,j}^{M}$ for $1\leq j\leq{N\choose M}$ be the different elements of $\mathcal{R}_{k}^{M}$. Let $\hat{p}_{k}^{M}$ be the estimate of $p_{k}^{M}$, where they can be computed as

[TABLE]

In the case of dependent arms, $r_{k,j}^{M}$’s are defined in the same way as independent arms, but note that the set $S_{\mathcal{K}}$ corresponding to $r_{k,j}^{M}$ is used for generating $r_{k^{\prime},j}^{M}$ for all $k^{\prime}\in\mathcal{K}$. Hence, $\hat{p}_{k}^{M}$ is defined as follows for dependent arms:

[TABLE]

The FTE-MAB algorithm selects arm $\hat{k}=\underset{k}{\operatorname*{argmax}}\ \hat{p}_{k}^{M}$ for $M$-time exploitations. This algorithm is summarized in Algorithm 2. We next present a theorem for an upper bound of the minimum number of experiments needed to guarantee an upper bound on regret of Algorithm 2 which is the generalization of Theorem 1.

Theorem 2

For any $0<\epsilon_{r},\Delta p<1,$ if each of the $K$ arms is explored for $N$ times in the experimentation phase such that $\lfloor\frac{N}{M}\rfloor\geq\frac{2\ln\left(\frac{2K}{\epsilon_{r}}\right)}{{\Delta p}^{2}}$, the finite-time exploitation regret defined in Definition 2 is bounded by $\epsilon_{r}$, i.e. $r_{M}(\Delta p)\leq\epsilon_{r}$. If the rewards of different arms are dependent, simultaneous explorations of the $K$ arms are required for the same bound on regret.

The proof of Theorem 2 is similar to that of Theorem 1, which can be found in the Appendix.

Let $p_{(1)}^{M},p_{(2)}^{M},\dots,p_{(K)}^{M}$ be the ordered list of $p_{1}^{M},$ $p_{2}^{M},\dots,p_{K}^{M}$ in descending order. Note that arm $(1)$ is actually arm $k^{*}$ defined in Equation (1). Define the difference between the two maximum $p_{k}^{M}$’s as $\Delta p^{*}_{M}=p_{(1)}^{M}-p_{(2)}^{M}$, where without loss of generality is assumed to be nonzero. Having the knowledge of $\Delta p^{*}_{M}$ or a lower bound on it, a stronger notion of regret can be defined as

[TABLE]

and have the following corollary.

Corollary 2

From the theoretical point of view, upon the knowledge of $\Delta p^{*}_{M}$ or a lower bound on it, for any $0<\epsilon_{r}<1$, the regret defined in Equation (14) is bounded by $\epsilon_{r},$ i.e. $r_{M}<\epsilon_{r}$, if the $K$ arms are explored for $N$ times each, where $\lfloor\frac{N}{M}\rfloor\geq\frac{2\ln\left(\frac{2K}{\epsilon_{r}}\right)}{{\Delta p^{*}_{M}}^{2}}$. If arms are dependent, instantaneous explorations of the $K$ arms are needed.

Corollary 3

If $M$ converges to infinity, the problem becomes the classical multi-armed bandit problem since $\underset{k}{\operatorname*{argmax}}\ P(R_{k}^{M}\geq\boldsymbol{R}_{-k}^{M})$ is the same as $\underset{k}{\operatorname*{argmax}}\ P\left(\frac{R_{k}^{M}}{M}\geq\frac{\boldsymbol{R}_{-k}^{M}}{M}\right)$ and due to the law of large numbers $\frac{R_{k}^{M}}{M}\rightarrow\mu_{k}$ as $M\rightarrow\infty$. Hence, the FTE-MAB algorithm selects the arm with maximum expected reward if the arm is going to be exploited for infinitely many times and the cumulative reward is desired to be maximized.

V Simulation Results

In this section, we report numerical simulations validating the theoretical results presented in this paper. We compare our proposed OTE-MAB algorithm with the Upper Confidence Bound (UCB) [35], ExpExp [23], and MaRaB [13] algorithms. Consider two arms with the reward distributions given in example 1. The regret defined in Equation (10) versus the number of pure explorations for each arm, $N$, is averaged over 100,000 runs. The result is plotted in Figure 1 and as is shown OTE-MAB outperforms the state-of-the-art algorithms for the purpose of risk-aversion in terms of the regret defined in this paper. Note that the UCB algorithm aims at selecting an arm that maximizes the expected received reward, but in example 1, the arm with higher expected reward is less probable to have the highest reward, which is why the UCB algorithm performs poorly in this example. However, in the following example where the arm that rewards more on expectation is also more probable to reward more, the UCB, ExpExp, and MaRaB algorithms perform as well as the OTE-MAB algorithm.

Example 2

Consider two arms with the following unknown independent reward distributions:

[TABLE]

where $\alpha$ and $\beta$ are constants so that the two probability distribution functions integrate to one.

Note that in example 2, $\mathds{E}[R_{1}]>\mathds{E}[R_{2}]$ and $P(R_{1}\geq R_{2})>0.5$. For this scenario, the regret defined in Equation (10) versus the number of pure explorations for each arm, $N$, averaged over 100,000 runs is plotted in Figure 2.

In another experiment, the multi-armed bandit is simulated for example 1 and the probability that the selected arm has the higher reward is calculated over 500,000 runs for different algorithms. The result is shown in Figure 3. This result confirms the motivation of our study on risk-averse finite-time exploitations in multi-armed bandits.

In the above comparison of OTE-MAB with state-of-the-art algorithms, three different choices of hyper-parameters for the ExpExp and MaRaB algorithms are tested and the best performance is presented. However, note that the performances of these algorithms depend on the choice of hyper-parameter. In Figure 4, the sensitivity of the performance of ExpExp algorithm with respect to the choice of hyper-parameter $\rho$ is depicted for example 1 and a third example where the variance of the best arm is larger than the variance of the arm with lower expected reward. The two plots are the averaged regret over 100,000 runs versus the value of $\rho$ for the ExpExp algorithm for two different multi-armed bandit problems when $N=100$. As depicted in Figure 4, a choice of $\rho$ can be good for one multi-armed bandit problem, but not good for another one. Due to our observations, the sensitivity of the MaRaB algorithm to its hyper-parameter can even be more complex. Figure 5 depicts the averaged regret over 100,000 runs versus the value of MaRaB hyper-parameter, $\alpha$, when $N=100$. This figure is plotted for example 1 and a fourth example where reward of the first arm has a truncated Gaussian distribution with mean three and variance two over the interval $[0,10]$ and the second arm is the same as the one in example 1.

In another experiment, we compare the minimum number of explorations needed to guarantee a bound on regret for two cases of one-time and two-time exploitations. Theorems 1 and 2 suggest that for given $K,\epsilon_{r}$, and $\Delta p^{*}=\Delta p^{*}_{M}$, the upper bound of minimum number of explorations needed for $M$-time exploitations to guarantee that the regret is bounded by $\epsilon_{r}$ is $M$ times that of one-time exploitation. We design two examples of two-armed bandits such that $\Delta p^{*}=\Delta p^{*}_{2}=0.28$ and plot the minimum number of explorations to guarantee bounded regret by $\epsilon_{r}$ in Figure 6. The dashed line is the plot of the OTE-MAB algorithm multiplied by two which is close to the one related to the FTE-MAB algorithm for two-armed bandits. This observation supports our theoretical results.

Taking a closer look at example 1, we note that the regret defined in (10) can generally be formulated as

[TABLE]

Deriving the regret from the above equation, the same regret is found as the one generated by simulation for the OTE-MAB algorithm that is plotted in Figure 1. In this paper, the experimentation is assumed to have zero cost, which is often a valid assumption. However, if experimentation is time-consuming, there is a cost to postpone the exploitation of the best identified arm. For example, for more experimentation, a patient receives medication by delay or an investor keeps his/her money on hold with zero interest, both of which incur costs. Let such a cost be formulated by an increasing function $C(.)$, where $C(N)$ is the incurred cost of $N$ experiments. Then, a trade-off between more exploration for higher accuracy of best-arm identification and lower incurred cost of experimentation emerges. Such a trade-off can be formalized by solving

[TABLE]

where $\alpha$ is the cost-regret trade-off and $\bar{r}^{*}$ is calculated by Equation (15) based on an estimation of $p_{k^{*}}$ which is updated after each experiment. Figure 7 plots $C(N)+\alpha\cdot r^{*}$ under example 1 for $C(N)=\frac{N}{5},\alpha=100$, and the real value of $r^{*}$. The rigorous analysis of (16) is postponed for future work [36] and is beyond the scope of this paper.

VI Conclusion and Future Work

The focus of this work is on application domains, such as personalized health-care and one-time investment, where an experimentation phase of pure arm exploration is followed by a given finite number of exploitations of the best identified arm. We show through an example that the arm with maximum expected reward does not necessarily maximize the probability of receiving the maximum reward. The OTE-MAB and FTE-MAB algorithms are presented in this paper whose goals are to select the arm that maximizes the probability of receiving the maximum reward. We define a new notion of regret for our problem setup and find an upper bound on the minimum number of experiments that should be done to guarantee an upper bound on regret. The cost of experimentation is assumed to be negligible in this paper, but if such an assumption is violated in an application domain, one can study the cost-regret trade-off as a promising future work in various deterministic and stochastic versions.