Exponential two-armed bandit problem

TL;DR

This paper analyzes the exponential two-armed bandit problem using Bayesian methods, deriving strategies and risks, and compares it with the Gaussian case, revealing similar limiting behaviors and implications for batch processing.

Contribution

It develops a Bayesian approach for exponential two-armed bandits and derives a PDE for the limiting case, showing equivalence with Gaussian bandits in the limit.

Findings

Exponential and Gaussian bandits have the same description in the limit.

Batch processing does not increase Bayesian risk compared to individual processing as data size grows.

Derived recursive Bayesian strategies and risks for exponential bandits.

Abstract

We consider exponential two-armed bandit problem in which incomes are described by exponential distribution densities. We develop Bayesian approach and present recursive equation for determination of Bayesian strategy and Bayesian risk. In the limiting case as the control horizon goes to infinity, we obtain the second order partial differential equation in the domain of "close distributions". Results are compared with Gaussian two-armed bandit. It turned out that exponential and Gaussian two-armed bandits have the same description in the limiting case. Since Gaussian two-armed bandit describes the batch processing, this means that in case of exponential two-armed bandit batch processing does not enlarge Bayesian risk in comparison with one-by-one optimal processing as the total number of processed data items goes to infinity.

Equations

$$\displaystyle f(x|m_{\ell})=\left\{\begin{array}[]{l}m_{\ell}^{-1}\exp(-xm_{\ell}^{-1}),\ \ x\geq 0,\\ 0,\qquad\qquad\qquad\qquad x<0,\end{array}\right.$$

$$L_N(\sigma ,\theta )=N\max (m_1,m_2)-{\mathrm{E}}_{\sigma ,\theta }\left(\sum _{n=1}^N{\xi }_n\right)$$

$$R_N(\mu )=\underset{\{\sigma \}}{\inf }{\int }_{\Theta }{L}_N(\sigma ,\theta )\mu (\theta )d\theta ,$$

$$R_N^M(\Theta )=\underset{\{\sigma \}}{\inf }\underset{\Theta }{\sup }{L}_N(\sigma ,\theta ),$$

$$R_N^M(\Theta )=R_N({\mu }_0)=\underset{\{\mu \}}{\sup }{R}_N(\mu ),$$

$$\mathrm{Pr}(y_{n+1}=\ell |X_1,n_1,X_2,n_2)={\sigma }_{\ell }(X_1,n_1,X_2,n_2),$$

$$\mu (m_1,m_2|X_1,n_1,X_2,n_2)=\frac{f(X_1,n_1|m_1)f(X_2,n_2|m_2)\mu (m_1,m_2)}{\mu (X_1,n_1,X_2,n_2)},$$

$$\mu (X_1,t_1,X_2,t_2)$$

$$={\iint }_{\Theta }f(X_1,n_1|m_1)f(X_2,n_2|m_2)\mu (m_1,m_2)d{m}_{1}d{m}_2.$$

$$\displaystyle f(X,n|m)=\left\{\begin{array}[]{l}\displaystyle{\frac{X^{n-1}}{m^{n}(n-1)!}}\exp(-Xm^{-1}),\ \ X\geq 0,\\ 0,\qquad\qquad\qquad\qquad\qquad\quad X<0,\end{array}\right.$$

$$R(X_1,n_1,X_2,n_2)=\min (R^{(1)}(X_1,n_1,X_2,n_2),R^{(2)}(X_1,n_1,X_2,n_2)),$$

$$R^{(1)}(X_1,n_1,X_2,n_2)=R^{(2)}(X_1,n_1,X_2,n_2)=0$$

$$R^{(1)}(X_1,n_1,X_2,n_2)$$

$$\displaystyle=\displaystyle{\iint_{\Theta}}\mu(m_{1},m_{2}|X_{1},n_{1},X_{2},n_{2})\times\Big{(}(m_{2}-m_{1})^{+}$$

$$\displaystyle+\displaystyle{\int_{0}^{\infty}}R(X_{1}+Y,n_{1}+1,X_{2},n_{2})f(Y|m_{1})dY\Big{)}dm_{1}dm_{2},$$

$$R^{(2)}(X_1,n_1,X_2,n_2)$$

$$\displaystyle=\displaystyle{\iint_{\Theta}}\mu(m_{1},m_{2}|X_{1},n_{1},X_{2},n_{2})\times\Big{(}(m_{1}-m_{2})^{+}$$

$$\displaystyle+\displaystyle{\int_{0}^{\infty}}R(X_{1},n_{1},X_{2}+Y,n_{2}+1)f(Y|m_{2})dY\Big{)}dm_{1}dm_{2}.$$

$$R_N(\mu )=R(0,0,0,0).$$

$$R_N(\mu )=n_0{\iint }_{\Theta }|m_2-m_1|\mu (m_1,m_2)d{m}_{1}d{m}_2$$

$$+{\iint }_0^{\infty }R(n_0,X_1,n_0,X_2)\mu (X_1,n_1,X_2,n_2)d{X}_{1}d{X}_2.$$

$$\tilde{R}(X_1,n_1,X_2,n_2)=R(X_1,n_1,X_2,n_2)\times \mu (X_1,n_1,X_2,n_2),$$

$$\tilde{R}(X_1,n_1,X_2,n_2)=\min ({\tilde{R}}^{(1)}(X_1,n_1,X_2,n_2),{\tilde{R}}^{(2)}(X_1,n_1,X_2,n_2)),$$

$${\tilde{R}}^{(1)}(X_1,n_1,X_2,n_2)={\tilde{R}}^{(2)}(X_1,n_1,X_2,n_2)=0$$

$${\tilde{R}}^{(1)}(X_1,n_1,X_2,n_2)=G^{(1)}(X_1,n_1,X_2,n_2)$$

$$+{\int }_0^{\infty }\tilde{R}(X_1+Y,n_1+1,X_2,n_2)\times \frac{n_1{X}_1^{n_1-1}}{(X_1+Y{)}^{n_1}}dY,$$

$${\tilde{R}}^{(2)}(X_1,n_1,X_2,n_2)=G^{(2)}(X_1,n_1,X_2,n_2)$$

$$+{\int }_0^{\infty }\tilde{R}(X_1,n_1,X_2+Y,n_2+1)\times \frac{n_2{X}_2^{n_2-1}}{(X_2+Y{)}^{n_2}}dY.$$

$$G^{(1)}(X_1,n_1,X_2,n_2)$$

$$={\iint }_{\Theta }(m_2-m_1{)}^{+}f(X_1,n_1|m_1)f(X_2,n_2|m_2)\mu (m_1,m_2)d{m}_{1}d{m}_2,$$

$$G^{(2)}(X_1,n_1,X_2,n_2)$$

$$={\iint }_{\Theta }(m_1-m_2{)}^{+}f(X_1,n_1|m_1)f(X_2,n_2|m_2)\mu (m_1,m_2)d{m}_{1}d{m}_2.$$

$$R_N(\mu )=\tilde{R}(0,0,0,0).$$

$$R_N(\mu )=n_0{\iint }_{\Theta }|m_2-m_1|\mu (m_1,m_2)d{m}_{1}d{m}_2$$

$$+{\iint }_0^{\infty }\tilde{R}(n_0,X_1,n_0,X_2)d{X}_{1}d{X}_2.$$

$$\mathrm{E}({\xi }_n||y_n=\ell )=m_{\ell },\mathrm{Var}({\xi }_n||y_n=\ell )=D_{\ell }=m_{\ell }^2,$$

$$m_{\ell }=m+(D/N{)}^{1/2}{v}_{\ell },$$

Full text

93E20,

62L05,

62C10,

62C20,

62F35,

Poissonian two-armed bandit,

Bayesian approach,

, and

1 Introduction

We consider the two-armed bandit problem (see, e.g. [1, 2]) in the following setting. Let $\xi_{n}$, $n=1,2,\dots,N$, be a controlled random process which values are interpreted as incomes, depend only on currently chosen actions $y_{n}\in\{1,2\}$ and are described by exponential distribution density

[TABLE]

if $y_{n}=\ell$, $\ell=1,2$. Here $m_{1},m_{2}$ are one-step mathematical expectations of income and a vector parameter $\theta=(m_{1},m_{2})$ completely describes exponential two-armed bandit. We assume that the set of admissible values of parameters $\Theta=\{\theta\}$ is a priori known.

A control strategy $\sigma$ generally assigns a random choice of the action at the point of time $n+1$ depending on currently observed history of the process. For exponential distributions of incomes the history consists of cumulative numbers $n_{1},n_{2}$ ($n_{1}+n_{2}=n$) of both actions applications and corresponding cumulative incomes $X_{1},X_{2}$. If one knew both $m_{1},m_{2}$, he should always choose the action corresponding to the largest of them, his total expected income on the control horizon $N$ would thus be equal to $N\max(m_{1},m_{2})$. But if he uses strategy $\sigma$, his total expected income is less than maximal by the value

[TABLE]

which is called the regret. Here $\mathrm{E}_{\sigma,\theta}$ denotes the mathematical expectation with respect to the measure generated by strategy $\sigma$ and parameter $\theta$.

Let’s assign a prior distribution density $\mu(\theta)=\mu(m_{1},m_{2})$ on the set of parameters $\Theta$. Corresponding Bayesian risk is defined as follows

[TABLE]

the optimal strategy $\sigma^{B}$ is called Bayesian strategy. Note that Bayesian approach allows to determine Bayesian risk and Bayesian strategy by solving recursive Bellman-type equation for arbitrary prior distribution. The minimax risk on the set $\Theta$ is defined as

[TABLE]

corresponding optimal strategy $\sigma^{M}$ is called minimax strategy. There is no a direct method of determining minimax strategy and minimax risk. However, one can determine them using the main theorem of the theory of games according to which the following equality holds

[TABLE]

i.e. minimax risk is equal to the Bayesian calculated with respect to the worst-case prior distribution and minimax strategy is equal to corresponding Bayesian strategy.

There are some different approaches to the two-armed bandit problem. We refer here to [3], [4], [5] and references therein.

The rest of the paper is the following. Recursive Bellman-type equation for determining Bayesian risk and Bayesian strategy is presented in Section 2. Another version of recursive equation is presented in Section 3. In Section 4, we obtain a second order partial differential equation in the limiting case as $N\to\infty$. In Section 5 we compare exponential and Gaussian two-armed bandits. It turned out that they have the same description in the limiting case. Since Gaussian two-armed bandit describes the batch processing (see, e.g. [5]), this means that in case of exponential two-armed bandit batch processing does not enlarge Bayesian risk in comparison with one-by-one optimal processing asymptotically as $N\to\infty$.

2 Recursive equation

Let’s consider control strategies $\{\sigma_{\ell}(X_{1},t_{1},X_{2},t_{2})\}$ which are defined by a condition

[TABLE]

where $n_{1},n_{2}$ are current cumulative times of both actions applications, $X_{1},X_{2}$ are corresponding current cumulative incomes. The posterior distribution at the point of time $n=n_{1}+n_{2}$ is calculated as

[TABLE]

where

[TABLE]

Here $f(X,n|m)$ is defined as

[TABLE]

Note that $f(X,1|m)=f(X|m)$ is exponential distribution density. Let’s put $f(0,0|m)=1$. Then (2.1) remains correct if $n_{1}=0$ and/or $n_{2}=0$. Denote $x^{+}=\max(x,0)$. Using (1.3), (1.4) we obtain the following recursive Bellman-type equation for determining Bayesian risk (1.5) with respect to the posterior distribution (2.1):

[TABLE]

where

[TABLE]

if $n_{1}+n_{2}=N$ and then

[TABLE]

Here $\{R^{(\ell)}(X_{1},n_{1},X_{2},n_{2})\}$ denote expected losses if initially the $\ell$-th action is applied at the point of time $n+1$ and then control is optimally implemented ($\ell=1,2$). Bayesian risk (1.5) is as follows

[TABLE]

Equation (2.3)–(2.5) allow to determine Bayesian strategy, too. Bayesian strategy prescribes to choose $\ell$-th action if $R^{(\ell)}(X_{1},n_{1},X_{2},n_{2})$ has smaller value. In case of a draw $R^{(1)}(X_{1},n_{1},X_{2},n_{2})=R^{(2)}(X_{1},n_{1},X_{2},n_{2})$ the choice of the action may be arbitrary.

Given $N$ large enough, consider the strategy which at the start of the control $2n_{0}$ times equally applies both actions and then optimally controls. In this case

[TABLE]

3 One more version of recursive equation

In this section, we obtain another version of recursive Bellman-type equation. Let’s denote

[TABLE]

where $\{R(X_{1},n_{1},X_{2},n_{2})\}$ are Bayesian risks calculated with respect to the posterior distribution (2.1) and $\{\mu(X_{1},n_{1},X_{2},n_{2})\}$ are defined in (2.2). Then the following recursive equation holds

[TABLE]

where

[TABLE]

if $n_{1}+n_{2}=N$ and then

[TABLE]

Here

[TABLE]

Bayesian strategy prescribes to choose $\ell$-th action if $\tilde{R}^{(\ell)}(X_{1},n_{1},X_{2},n_{2})$ has smaller value. In case of a draw the choice of the action is arbitrary. Bayesian risk (1.5) is calculated by the formula

[TABLE]

Given $N$ large enough, consider the strategy which at the start of the control $2n_{0}$ times equally applies both actions and then optimally controls. In this case

[TABLE]

Formulas (3.1)–(3.6) follow from (2.3)–(2.7).

4 A limiting description

In this section, we present a limiting description by the second order partial differential equation. We consider the domain of “close distributions”, satisfying condition $|m_{1}-m_{2}|\leq cN^{-1/2}$ with $c$ large enough but independent from $N$, because just in this domain the maximum expected losses take place. Denote $\varepsilon=N^{-1}$, $\delta=N^{-1/2}$, so that $\delta^{2}=\varepsilon$. Note that one-step expected income and variance of exponential two-armed bandit are the following

[TABLE]

$\ell=1,2$. In the domain of distributions such that $m_{1},m_{2}$ are close to $m$, let’s put

[TABLE]

where $D=m^{2}$, $|x_{\ell}|\ll N^{1/2}$, $|y|\ll N^{1/2}$; $\ell=1,2$. Let’s estimate functions in (3.4). If $n_{\ell}$ is large enough then according to central limit theorem we have

[TABLE]

Hence, for functions in (3.4) one derives

[TABLE]

where

[TABLE]

Let’s estimate factors in integrals of (3.3). First, one derives

[TABLE]

with

[TABLE]

Let’s now estimate $f_{\ell}(1+y)$. Since

[TABLE]

and $\varepsilon(1+y)^{2}\ll 1$, we obtain that

[TABLE]

So, one can verify that

[TABLE]

Let’s put

[TABLE]

Using (4.1)–(4.4) and (4.6), one derives from (3.1)–(3.4) the integro-difference equation

[TABLE]

where

[TABLE]

if $t_{1}+t_{2}=1$ and then

[TABLE]

Bayesian strategy prescribes to choose $\ell$-th action if $\tilde{r}^{(\ell)}(x_{1},n_{1},x_{2},n_{2})$ has smaller value. In case of a draw the choice of the action is arbitrary. For the strategy which at the start of the control equally applies both actions $2n_{0}=2\varepsilon_{0}N$ times and then optimally controls, one derives that Bayesian risk (1.5) is calculated by the formula

[TABLE]

Formula (4.10) follows from (3.6) with the use of (4.1) and (4.6).

Finally, let’s present a limiting description of (4.9) by the second order partial differential equation. It is sufficient to consider the first equation of (4.9). The estimates below are carried out with accuracy to the terms of the order $\varepsilon$. First, we present $\tilde{r}(x_{1}+\delta y,t_{1}+\varepsilon,x_{2},t_{2})$ as Taylor series

[TABLE]

Substituting (4.11) into the first equation (4.9) and using (4.5) we obtain

[TABLE]

and, hence, in the limiting case as $\varepsilon\to 0$

[TABLE]

where $\tilde{r}=\tilde{r}(x_{1},t_{1},x_{2},t_{2})$. Similarly,

[TABLE]

Equations (4.12), (4.13) must be complemented by equation (4.7), which is now written as

[TABLE]

From (4.12)–(4.14) one derives

[TABLE]

Initial conditions are the following

[TABLE]

if $t_{1}+t_{2}=1$. Bayesian strategy prescribes to choose $\ell$-th action if $\ell$-th term in the left-hand side of (4.15) has smaller value. In case of a draw the choice of the action is arbitrary.

5 Comparison with Gaussian two-armed bandit

Gaussian two-armed bandit is characterized by incomes $\xi_{n}$, $n=1,2,\dots,N$, which values depend only on currently chosen actions $y_{n}\in\{1,2\}$ and are described by Gaussian (normal) distribution density

[TABLE]

if $y_{n}=\ell$, $\ell=1,2$. The variance $D$ is assumed to be known and expectations $m_{1},m_{2}$ are unknown. So, a vector parameter $\theta=(m_{1},m_{2})$ describes Gaussian two-armed bandit. We assume that the set of admissible values of parameters $\Theta=\{\theta\}$ is a priori known.

A control strategy $\sigma$ generally assigns a random choice of the action at the point of time $n+1$ depending on currently observed history $(X_{1},n_{1},X_{2},n_{2})$ where $n_{1},n_{2}$ ($n_{1}+n_{2}=n$) are cumulative times of both actions applications and $X_{1},X_{2}$ are corresponding cumulative incomes.

Again, one can assign a prior distribution density $\mu(m_{1},m_{2})$ and define a regret $L_{N}(\sigma,\theta)$ and Bayesian risk $R_{N}(\mu)$ just like in (1.4) and (1.5). To determine Bayesian risk in the domain of “close distributions” one should solve the following integro-difference equation (see, e.g. [5]):

[TABLE]

where

[TABLE]

if $t_{1}+t_{2}=1$ and then

[TABLE]

Here $f_{D}(x)=f_{D}(x|0)$ and $x_{1},x_{2},t_{1},t_{2},\varepsilon,\delta$ are defined in Section 4. Bayesian strategy prescribes to choose the $\ell$-th action if $\tilde{r}^{(\ell)}(x_{1},n_{1},x_{2},n_{2})$ has smaller value. In case of a draw the choice of the action is arbitrary. For the strategy which at the start of the control equally applies both actions $2n_{0}=2\varepsilon_{0}N$ times and then optimally controls, one derives that Bayesian risk (1.5) is calculated by the formula

[TABLE]

Here $\rho(v_{1},v_{2})$ is defined in Section 4. In the limiting case as $\varepsilon\to 0$, one can verify that integro-difference equation (5.4) results in the second order partial differential equation (4.15) with initial conditions (4.16). Recall that Gaussian two-armed bandit describes the batch processing [5]. So, this means that in case of exponential two-armed bandit batch processing does not enlarge Bayesian risk in comparison with one-by-one optimal processing asymptotically as $N\to\infty$.