Gittins' theorem under uncertainty

TL;DR

This paper extends Gittins' theorem to uncertain environments using nonlinear expectations, demonstrating that a Gittins index remains optimal under certain independence and relaxation conditions, with insights into exploration and uncertainty aversion.

Contribution

It introduces a framework for applying Gittins' theorem under uncertainty using nonlinear expectations and explores the impact of uncertainty on optimal decision-making.

Findings

Gittins index remains optimal under strong independence and relaxed optimality conditions.

Uncertainty influences the exploration-exploitation trade-off.

Numerical example illustrates the effect of uncertainty aversion on decisions.

Abstract

We study dynamic allocation problems for discrete time multi-armed bandits under uncertainty, based on the the theory of nonlinear expectations. We show that, under strong independence of the bandits and with some relaxation in the definition of optimality, a Gittins allocation index gives optimal choices. This involves studying the interaction of our uncertainty with controls which determine the filtration. We also run a simple numerical example which illustrates the interaction between the willingness to explore and uncertainty aversion of the agent when making decisions.

Equations

$$\gamma^{(m)}(s)=\operatorname*{ess\,inf}_{\tau\in\mathcal{T}^{(m)}(s)}\frac{\mathbb{E}^{\mathbb{P}^{(m)}}\big{(}\sum_{t=1}^{\tau}\beta^{t}h^{(m)}(s+t)\big{|}\mathcal{F}^{(m)}_{s}\big{)}}{\mathbb{E}^{\mathbb{P}^{(m)}}\big{(}\sum_{t=1}^{\tau}\beta^{t}\big{|}\mathcal{F}^{(m)}_{s}\big{)}}$$

$$\bar{\Omega }:=\prod _{m=1}^M{\Omega }^{(m)},\bar{\mathbb{P}}:=\underset{m=1}{\overset{M}{⨂}}{\mathbb{P}}^{(m)},\mathcal{F}(s):=\underset{m=1}{\overset{M}{⨂}}{\mathcal{F}}_{s^{(m)}}^{(m)}:s=(s^{(1)},...,s^{(M)})\in \mathcal{S}$$

$${\mathcal{T}}^{(m)}:=\left\{({\mathcal{F}}_t^{(m)}{)}_{t\ge 0}\text{-stopping times}\right\}$$

$$\mathfrak{T}(\mathcal{S}):=\left\{S=(S^{(1)},...,S^{(M)}):S^{(m)}\in {\mathcal{T}}^{(m)}\right\}.$$

$${\gamma }^{(m)}({\tilde{\eta }}^{(m)}(n))=\underset{k\in \mathcal{M}}{\min }{\gamma }^{(k)}({\tilde{\eta }}^{(k)}(n))\text{on each event}\{\tilde{\eta }(n+1)=\tilde{\eta }(n)+e^{(m)}\}$$

$$\inf_{\tilde{\eta}\in\mathcal{A}}\mathbb{E}^{\bar{\mathbb{P}}}\Big{(}\sum_{n=1}^{\infty}\sum_{m\in\mathcal{M}}\beta^{n}h^{(m)}(\tilde{\eta}^{(m)}(n))\big{[}\tilde{\eta}^{(m)}(n)-\tilde{\eta}^{(m)}(n-1)\big{]}\Big{)}.$$

$$\inf_{\tilde{\eta}\in\mathcal{A}}\mathbb{E}^{\bar{\mathbb{P}}}\Big{(}\sum_{n=1}^{\infty}\beta^{n}h^{(\rho_{n-1})}(t^{\rho}_{n})\Big{)}\;\;\;\text{where}\;\;\;t^{\rho}_{n}:=\sum_{k=0}^{n-1}\mathbb{I}(\rho_{k}=\rho_{n-1}).$$

$$V^{(m)}(i)=\min\bigg{(}\tilde{h}^{(m)}(i)+\beta\sup_{P\in\mathcal{U}^{(m)}}\sum_{j\in\mathcal{X}^{(m)}}P_{ij}V^{(m)}(j),\quad\frac{\gamma}{1-\beta}\bigg{)},\qquad i\in\mathcal{X}^{(m)}.$$

$$\tilde{\gamma}^{(m)}(i)=\inf_{\tau\in\mathcal{T}^{(m)},\tau\geq 1}\;\;\sup_{{\mathbb{Q}}\in\mathcal{Q}^{(m)}}\frac{\mathbb{E}^{\mathbb{Q}}\big{(}\sum_{t=1}^{\tau}\beta^{t}\tilde{h}^{(m)}(X^{(m)}_{t})\big{|}X^{(m)}_{0}=i\big{)}}{\mathbb{E}^{\mathbb{Q}}\big{(}\sum_{t=1}^{\tau}\beta^{t}\big{|}X^{(m)}_{0}=i\big{)}}.$$

$$V(i_{1},...,i_{K})=\min_{k\in[K]}\bigg{(}\tilde{h}^{(k)}(i_{k})+\beta\sup_{P\in\mathcal{U}^{(k)}}\sum_{j\in\mathcal{X}^{(k)}}P_{i_{k}j}V(i_{1},...,i_{k-1},j,i_{k+1},...,i_{K})\bigg{)}$$

$$0\le h^{(m)}(t)\le C^{(m)}\text{and}{h}^{(m)}(t)\to C^{(m)}\text{as}t\to \infty \mathbb{P}\text{-a.s.}$$

$$\mathcal{E}\left(\cdot |{\mathcal{F}}_t\right):L^{\infty }(\mathbb{P},{\mathcal{F}}_{\infty })\to L^{\infty }(\mathbb{P},{\mathcal{F}}_t)$$

$$\mathcal{E}\big{(}X\mathbb{I}_{A}+Y\mathbb{I}_{A^{c}}\big{|}\mathcal{F}_{t}\big{)}=\mathbb{I}_{A}\mathcal{E}\big{(}X\big{|}\mathcal{F}_{t}\big{)}+\mathbb{I}_{A^{c}}\mathcal{E}\big{(}Y\big{|}\mathcal{F}_{t}\big{)}.$$

$$\displaystyle\mathcal{E}\big{(}\cdot\big{|}\mathcal{F}_{\tau}\big{)}\;:L^{\infty}(\mathcal{F}_{\infty})\longrightarrow L^{\infty}(\mathcal{F}_{\tau}),\qquad X\longmapsto\sum_{t=0}^{\infty}\mathbb{I}(\tau=t)\;\mathcal{E}\big{(}X\big{|}\mathcal{F}_{t}\big{)}.$$

$$\mathcal{E}\big{(}\cdot\big{|}\mathcal{F}_{\tau}\big{)}=\operatorname*{ess\,sup}_{\mathbb{Q}\in{\mathcal{Q}}}\mathbb{E}^{\mathbb{Q}}\big{(}\cdot\big{|}\mathcal{F}_{\tau}\big{)}$$

$$\mathcal{E}^{(m)}\Big{(}\;\cdot\;\Big{|}\mathcal{F}^{(m)}_{S^{(m)}}\Big{)}=\operatorname*{ess\,sup}_{\mathbb{Q}\in\mathcal{Q}^{(m)}}\mathbb{E}^{\mathbb{Q}}\Big{(}\;\cdot\;\Big{|}\mathcal{F}^{(m)}_{S^{(m)}}\Big{)}$$

$${\mathfrak{E}}_S:L^{\infty }\left(\bar{\Omega },\bar{\mathbb{P}},\mathcal{F}(\infty )\right)$$

$$X$$

$${\mathcal{Q}}:=\bigg{\{}\bigotimes_{m=1}^{M}\mathbb{Q}^{(m)}\;\;\text{for}\;\;\mathbb{Q}^{(m)}\in\mathcal{Q}^{(m)}\bigg{\}}.$$

$${\mathfrak{E}}_S\left(\cdot \right)\le {\mathfrak{E}}_S\left({\mathfrak{E}}_{S^{\prime }}\left(\cdot \right)\right)\bar{\mathbb{P}}\text{-a.s.}$$

$$Y({\omega }^{(1)},...,{\omega }^{(M)})=X^{(1)}({\omega }^{(1)})\times \cdots \times X^{(M)}({\omega }^{(M)})\bar{\mathbb{P}}\text{-a.s.},$$

$$\mathfrak{E}_{S}(Y)=\mathcal{E}^{(1)}\left(X^{(1)}\Big{|}\mathcal{F}^{(1)}_{S^{(1)}}\right)\times\cdots\times\mathcal{E}^{(M)}\left(X^{(M)}\Big{|}\mathcal{F}^{(M)}_{S^{(M)}}\right)\;\;\;\bar{\mathbb{P}}\text{-a.s.}$$

$$\mathfrak{E}_{S}(\tilde{X})=\mathcal{E}^{(m)}\big{(}X\big{|}\mathcal{F}^{(m)}_{S^{(m)}}\big{)}\;\;\;\bar{\mathbb{P}}\text{-a.s.}$$

$$\displaystyle\sup_{\mathbb{Q},\tilde{\mathbb{Q}}}\mathbb{E}^{\mathbb{Q}\otimes\tilde{\mathbb{Q}}}\big{(}f(X,\tilde{X})\big{)}$$

$$\displaystyle=\max\Big{(}\frac{f(0,0)+f(0,1)}{2},\frac{f(1,0)+f(1,1)}{2}\Big{)}$$

$$\displaystyle\sup_{\tilde{\mathbb{Q}}}\mathbb{E}^{\tilde{\mathbb{Q}}}\Big{(}\sup_{{\mathbb{Q}}}\mathbb{E}^{{\mathbb{Q}}}\big{(}f(X,\tilde{x})\big{)}\Big{|}_{\tilde{x}=\tilde{X}}\Big{)}=\frac{\max\{f(0,0),f(1,0)\}}{2}+\frac{\max\{f(0,1),f(1,1)\}}{2}.$$

$$\mathfrak{E}\Big{(}\mathfrak{E}_{(0,1)}\Big{(}f(X,\tilde{X})\Big{)}\Big{)}\neq\mathfrak{E}\Big{(}\mathfrak{E}_{(1,0)}\Big{(}f(X,\tilde{X})\Big{)}\Big{)}.$$

$$\mathcal{G}^{\rho}_{n}:=\Big{\{}A\in\mathcal{F}(T)\;:\;A\cap\{\tilde{\eta}(n)=r\}\in\mathcal{F}(r)\;\;\;\forall r\in\mathcal{S}\Big{\}},$$

$$\mathfrak{E}\bigg{(}\mathbb{I}_{A}\bigg{(}\sum_{n=N+1}^{\infty}g^{\rho^{*}}(n)\bigg{)}\bigg{)}\leq\mathfrak{E}\bigg{(}\mathbb{I}_{A}\bigg{(}\sum_{n=N+1}^{\infty}g^{\rho}(n)\bigg{)}\bigg{)}\qquad\text{for all }A\in\mathcal{G}^{\rho}_{N}(=\mathcal{G}^{\rho^{*}}_{N}).$$

$$\mathcal{E}\bigg{(}\sum_{n=N+1}^{\infty}g^{\rho^{*}}(n)\bigg{|}\mathcal{F}_{N}\bigg{)}=\operatorname*{ess\,inf}_{\rho\sim_{N}\rho^{*}}\mathcal{E}\bigg{(}\sum_{n=N+1}^{\infty}g^{\rho}(n)\bigg{|}\mathcal{F}_{N}\bigg{)}.$$

$$V_{N}=\operatorname*{ess\,inf}_{\rho}\mathcal{E}\bigg{(}\sum_{n=N+1}^{\infty}g^{\rho}(n)\bigg{|}\mathcal{F}_{N}\bigg{)}.$$

Full text

Keywords: Gittins index, uncertainty, nonlinear expectation, multi-armed bandits, time-consistency, robustness.

MSC 2010: 93E35, 60G40, 91B32, 91B70

1 Introduction

When making decisions, people generally have a strict preference for options which they understand well. Since the classical work of Knight [52] and Keynes [50], there has been a stream of thinking within economics and statistics that focuses on the difference between the randomness of an outcome and lack of knowledge of its probability distribution (sometimes called ‘Knightian uncertainty’). This lack of knowledge is often related to estimation, as the probabilities used are often based on past observations.

This raises a natural question: how should we make decisions, given they will affect both our short-term outcomes, and the information available in the future? Shall we make a decision to explore and obtain new information, or shall we exploit the information available to optimize our profit? A simple setting in which this arises is a multi-armed bandit problem.

Modeling learning of the distribution of outcomes leads us to a paradox due to inconsistency in our decisions. As Keynes is said to have remarked333It appears this quote may be misattributed. One suggestion (discussed by John Kay [48]) is that the correct attribution is to Paul Samuelson, and should read “When my information changes, I alter my conclusions”, which fits even more easily with the thrust of this paper. , “When the facts change, I change my mind. What do you do, sir?” The question a rational decision maker faces is, “if I suspect that I will change my opinions or preferences tomorrow, how do I account for this today?”

In this paper, we use the theory of nonlinear expectations (or equivalently risk measures) which are known to model Knightian uncertainty (see Föllmer and Schied [36]). This has been used to address statistical uncertainty, for example in [23, 25]. To achieve consistency in decision making, we usually have to consider a time-consistent nonlinear expectation. This is widely studied through backward stochastic differential equations (BSDEs) (see, for example, the work of Peng and others [62, 61, 63, 59]).

However, these approaches presume that the flow of information is not controlled. (Formally, the filtration of our agent is fixed and independent of their controls.) When we can control the observations which we will receive, this is not the case. In order to address this issue, while accounting for uncertainty, we discuss an alternative approach to deriving a time-consistent control problem, based on ideas from indifference pricing and the martingale optimality principle. Using this approach, we show that when comparing different independent options, we can calculate an index separately for each alternative such that the ‘optimal’ strategy is always to choose the option with the smallest index. This idea was initially proposed by Gittins and Jones [41] (see also [42, 40]) in a context where the probability measure is fixed but estimation (in a Bayesian perspective) is modeled by the evolution of a Markov process.

Given this result, we demonstrate a numerical solution in a simple setting. We shall see that our algorithm gives behaviour which is both optimistic and pessimistic in different regimes, and compares well with existing methods for multi-armed bandits.

1.1 Multi-armed bandits

Multi-armed bandits are a classical toy example with which to study decision making with randomness. They are commonly known to have applications in medical trials (Armitage [4] or Anscombe [3]) and experimental design (Berry and Fristedt [13] or the classic paper of Robbins [69]), along with other areas. A few recent works in finance for portfolio selection can also be found in Huo and Fu [45] or Shen et al. [72]. The basic idea is that one has $M$ ‘bandits444‘One-armed bandits’ are an early variety of automated gambling system, the descendants of which are also known as slot machines, fruit machines or poker machines, depending on nationality.’, or equivalently, a bandit with $M$ arms, and one must choose which bandit should be played at each time. A key paper studying these systems, Gittins and Jones [41], argued that for a collection of independent bandits, each governed by a countable state Markov process, one could compute the “Gittins index” for each bandit separately, and the optimal strategy is to play the machine with the lowest index (or the highest, depending on the sign of gains/losses). The proof of this result has been obtained using a number of different perspectives, for example Weber’s prevailing charge formulation [75] (which we consider in more detail below), Whittle’s retirement option formulation [76] and its extension without a Markov assumption by El Karoui and Karatzas [32] (and [33] in continuous time). A review of the proofs in discrete time is given by Frostig and Weiss [39]. However, in all these cases, the objective to be optimized is the discounted expected gain/loss – in particular, we are assumed to have no risk-aversion or uncertainty-aversion.

Gittins’ index theory is commonly known as the first solution to an adaptive and sequential experimental design problem (from a Bayesian perspective) where the payoff of each bandit is assumed to be generated from a fixed unknown distribution555There are a few variations on these assumption e.g. adversarial bandits, contextual bandits or non-stationary bandits. Reviews of these can be found in Burtini et al. [19] and Zhou [80]. which must be inferred ‘on-the-fly’, but where experimentation may be costly. As an alternative to Gittins’ index, Agrawal [1] proposed the ‘Upper Confidence Bound (UCB) algorithm’ which achieves a regret (deviation of average reward from the optimal reward) with the minimal asymptotic order of $\log(N)$, as proved by Lai and Robbins [54]. In the UCB algorithm, we compute a confidence interval for the expected reward at each step, and then play the bandit with the largest upper bound (where positive outcomes are preferred). Intuitively, using an upper bound encourages us to try bandits where we are less certain of the average reward, which encourages exploration. This is a form of ‘optimism’ in decisions, which is counter-intuitive from the classical ‘pessimistic’ utility theory (à la von Neumann and Morgenstern [74]), where our preferences are for more certain outcomes.

Typically, under appropriate assumptions, it is also the case that Gittins’ index is a form of upper confidence bound for the estimated reward, an idea originally based on observations in Bather [12] and Kelly [49] and explored in more detail by Chang and Lai [21], followed up by Brezzi and Lai [18] (see Yao [78] for an error correction). Lattimore [55] also proves that Gittins’ index achieves a minimal order bound on regret.

The apparent contradiction between the optimism of the UCB algorithm and Gittins’ index and the pessimism of classical utility theory is what led to this paper. We extend the notion of Gittins’ index to a robust (nonlinear) operator, allowing for uncertainty aversion. We work in a generic discrete-time setting, allowing for the possibility of online learning, non-stationary and continuous outcomes, embedding all these effects in an abstract ‘nonlinear expectation’. (A concrete application to a simple setting with learning and uncertainty is given in Section 6.) In particular, we reformulate the proof of Gittins index theorem proposed by Weber [75], as this proof relies the least on the linearity of the expectation, and gives a natural form of time-consistency. We also remove a Markov assumption in Weber’s proof by adapting El Karoui and Karatzas’ formulation [32]. Our solution involves an optimal stopping problem under a nonlinear expectation, which can be converted to a low dimensional reflected BSDE (see for example El Karoui et al. [31] and Cheng and Riedel [22] in continuous time or An, Cohen and Ji [2] in discrete time). This allows us to see a balance between the desire to explore and to exploit in our decision making.

The robust version of Gittins index has some correlation to the adversarial bandit problem (see, for example, Auer et al. [7, 8]) where we are playing the bandit against an adversary. Our theory proposes an ‘optimal’ deterministic strategy (no additional randomness is introduced at the decision time) against an adversary who tries to maximize our cost, which is slightly different from the known random algorithms for the adversarial bandit problem. The key difference is that, in the classical adversarial problem, an adversary is trying to maximize our ‘regret’ whereas in our setting, we view the adversary as trying to maximize our cost. In our setting, the adversary is also permitted to respond to our current controls at every time, and we do not assume a minimax theorem holds.

The study of an adversary for the payoff in the bandit problem (via Gittins index) has been considered by Caro and Gupta [20] and Kim and Lim [51] (with additional penalty in the reward) using Whittle’s retirement option argument [76]. In their works, they rely heavily on a Markov assumption, which allows them to postulate a robust dynamic programming principle (see also Iyengar [46], Nilim and El Ghaoui [58]). Their formulation considers the robust Gittins’ strategy as a promising solution due to its optimality for a single bandit, but they do not show optimality for multiple bandits. Furthermore, their Markov assumption restricts them to have a fixed uncertainty at all times and, therefore, it is not clear how to incorporate learning in their model. In contrast, our framework pays more attention to defining a good notion of dynamic optimality for our nonlinear expectation without any Markov assumption.

By encoding learning through nonlinear expectations, a wide range of modeling options are included in our approach. For example, statistical concerns are treated in this framework in [23, 24] or Bielecki, Cialenco and Chen [14]. We could also allow adversarial choices with a range of a fixed set (as in the classical adversarial bandit problem [7] or as in [20, 51]) or a random set which can be used to model learning as in the classical Gittins’ theory. We also allow dynamic adversaries, which are not considered in the usual adversarial setting.

The paper proceeds as follows: In Section 2, we present some relevant existing approaches to multi-armed bandits, which we will adapt and combine to obtain our result. In Section 3, we give the required definitions for the nonlinear expectations that we use to evaluate our decisions. We also discuss the different notions of optimality which are available, and how they interact with the dynamic programming principle.

In Section 4, we give a summary of how we apply these expectations to a multi-armed bandit problem, state the key result, and give a sketch outline of the proof. The full details of this (rather technical) proof are given in two appendices: Appendix A works through the first half of the proof, giving careful analysis of an optimal stopping problem under nonlinear expectation, and the corresponding ‘fair value’ process, for a single bandit; Appendix B gives the second half of the proof, and demonstrates that the single bandit analysis yields an optimal strategy when deciding between multiple bandits. Further technical lemmas, which are used but do not contributed significantly to the main proof, are given in Appendix C.

Section 5 considers a simple example of a multiple bandit problem numerically, suggesting some connections with behavioural finance; the algorithm used to compute this example is given in Appendix D.

2 Problem formulation and related approaches

2.1 General Problem Formulation

Broadly speaking, Gittins [41] argues that in order to dynamically allocate a single resource amongst several alternative projects, the optimal policy is to play at each point a bandit of lowest “Gittins’ index”. This index can be computed separately for each bandit, by solving an optimal stopping problem.

The subtlety in the proof of Gittins’ theorem is to give a tractable representation of the class of control policies available to the decision maker. In the original formulation (see, for example, [41, 75, 76, 39]), the class considered is feedback controls, as in a standard Markovian stochastic control problem; i.e. the system of bandits is modelled as a single Markov process, and the controls alter its transition probabilities. This formulation is restrictive, as it is not clear how it can be applied to a non-Markovian framework. Furthermore, as the control determines the filtration observed, it is also not clear how to introduce a general form of uncertainty aversion in this framework.

El Karoui and Karatzas [32] extend the argument of Gittins’ theorem to the general case, without a Markovian assumption, by using Mandelbaum’s [57] “allocation strategy” formulation of the class of control policies. In particular, they view the cost of the bandits as a fixed process. The effect of allocation is to delay the realization of these fixed costs, which results in a benefit to the decision maker due to the time-value of money.

In this paper, we will use a slight modification of Mandelbaum’s allocation strategy to describe optimal strategies using a robust Gittins’ theorem (without a Markovian assumption) via the classical argument given by Weber [75], but using El Karoui and Karatzas’ [32] formulation.

Remark 1*.*

In most of the literature on Gittins’ theorem, maximization of rewards is usually considered. For convenience, as is common in the theory of nonlinear expectations, we will consider the minimization of costs instead. Our presentation of others’ results is done with the corresponding changes in sign.

Assumption 2.1**.**

Suppose that we have $M$ bandits. The $m$th bandit is associated with a filtered probability space \big{(}\Omega^{(m)},\mathbb{P}^{(m)},(\mathcal{F}^{(m)}_{t})_{t\geq 0}\big{)}. Playing this bandit for the $t$th time realizes a non-negative bounded cost $h^{(m)}(t)$, where the process \big{(}h^{(m)}(t)\big{)}_{t\geq 1} is adapted to $(\mathcal{F}^{(m)}_{t})_{t\geq 0}$. We assume that $\mathcal{F}^{(m)}_{0}=\{\phi,\Omega^{(m)}\}$.

The goal of the decision maker is to minimize the discounted total cost, for a given discount factor $\beta\in(0,1)$, when they can choose the order in which bandits are played.

Before considering a robust approach, we first outline the solution to this problem in a standard setting of classical expectation.

Definition 2.2**.**

The Gittins index at time $s\geq 0$ of the $m$th bandit is given by

[TABLE]

where $\mathcal{T}^{(m)}(s)$ is the space of positive $(\mathcal{F}^{(m)}_{s+t})_{t\geq 0}$-stopping times666Equivalently, for $\tau\in\mathcal{T}^{(m)}(s)$, $s+\tau$ is an $(\mathcal{F}^{(m)}_{t})_{t\geq 0}$-stopping time. and the essential infimum is taken in $L^{\infty}(\mathcal{F}_{s}^{(m)})$.

Definition 2.3**.**

We define the orthant probability space $\left(\bar{\Omega},\bar{\mathbb{P}},\left(\mathcal{F}(s)\right)_{s\in\mathcal{S}}\right)$ by

[TABLE]

where $\mathcal{S}:=\mathbb{N}_{0}^{M}$. We write $\mathcal{F}(\infty):=\bigotimes_{m=1}^{\infty}\mathcal{F}^{(m)}_{\infty}$.

We call $\left(\mathcal{F}(s)\right)_{s\in\mathcal{S}}$, the orthant filtration.

To describe a useful set of stopping times in this (multi-indexed) filtration, let

[TABLE]

and

[TABLE]

For $S\in\mathfrak{T}(\mathcal{S})$, we write $\mathcal{F}(S):=\bigotimes_{m=1}^{M}\mathcal{F}^{(m)}_{S^{(m)}}\quad\text{for }S\in\mathfrak{T}(\mathcal{S}).$

2.1.1 Classical Gittins Theorem

Our policies will be described by a (random) path in the space $\mathcal{S}$, which indicates how many times each bandit has been played.

Definition 2.4** (Mandelbaum [57]).**

The Mandelbaum allocation strategy is an $\mathcal{S}$-valued random sequence \big{(}\tilde{\eta}(n)\big{)}_{n\geq 0} such that

1. (i)

$\tilde{\eta}(0)=0$ 2. (ii)

$\tilde{\eta}(n+1)=\tilde{\eta}(n)+e^{(m)}$ for some $m\in\mathcal{M}=:\{1,...,M\}$. 3. (iii)

$\{\tilde{\eta}(n+1)=\tilde{\eta}(n)+e^{(m)},\;\tilde{\eta}(n)=r\}\in\mathcal{F}(r)$ for all $m\in\mathcal{M}:=\{1,...,M\}$ and for all $r\in\mathcal{S}$.

Here, $e^{(m)}$ denotes the $m$th unit vector in $\mathcal{S}$. We denote by $\mathcal{A}$ the family of all Mandelbaum allocation strategy.

Theorem 2.5** (Gittins’ theorem, as proved by El Karoui and Karatzas [32]).**

Let \big{(}\tilde{\eta}^{*}(n)\big{)}_{n\geq 0} be a Mandelbaum allocation strategy \big{(}\tilde{\eta}^{*}(n)\big{)}_{n\geq 0} such that

[TABLE]

for all $m\in\mathcal{M}$ and $n\geq 0$. Then \big{(}\tilde{\eta}^{*}(n)\big{)}_{n\geq 0} is an optimal solution to the optimisation problem

[TABLE]

In particular, Theorem 2.5 says that the strategy which always plays the bandit with the minimum index minimizes the expectation of the total discounted cost.

Remark 2*.*

A Mandelbaum allocation strategy \big{(}\tilde{\eta}(n)\big{)}_{n\geq 0} can also be represented by its increments, in particular, by a sequence of decision variables $(\rho_{n})_{n\geq 0}$ taking values in $\mathcal{M}$. In other words, we can define $(\rho_{n})_{n\geq 0}$ such that $\{\rho_{n}=m\}=\{\tilde{\eta}(n+1)=\tilde{\eta}(n)+e^{(m)}\}$. We may then replace the objective equation (2.1) by

[TABLE]

Remark 3*.*

Our paper considers the orthant filtration as the product of filtrations defined on different spaces. This is slightly different from Mandelbaum [57] (and thus El Karoui and Karatzas [32]) where the orthant filtration is considered as the join of filtrations defined on the same space. This technical difference will allow us to more easily define a ‘Nonlinear expectation’ which still carries some form of independence and ‘time-consistency’. (See discussion in Section 3.2.)

Remark 4*.*

In El Karoui and Karatzas [32], it is assumed that the cost process is predictable, instead of adapted, with respect to the filtration of the bandit. When using a classical expectation, there is no modelling difference between predictable and adapted cost processes (as one can just take the conditional expectation to reduce adapted costs to predictable costs). However, under a ‘nonlinear expectation’, this is not the case, so we give the more general result with adapted costs.

2.1.2 Robust Gittins Index

Under a Markovian assumption, Caro and Gupta [20] consider a ‘robust’ Gittins index based on the Robust Bellman equation studied in Iyergar [46] and Nilim and El Ghaoui [58]. (Similar work is considered by Kim and Lim [51] with an additional penalty in the formulation.)

The following assumptions are used in Caro and Gupta [20] (translated into our notation):

1. (i)

The cost process is driven by some underlying finite-state process $(X^{(m)}_{t})_{t\geq 0}$ taking values in $\mathcal{X}^{(m)}$. i.e. we have $h^{(m)}(t)=\tilde{h}^{(m)}(X^{(m)}_{t})$ for some deterministic function $\tilde{h}^{(m)}$. 2. (ii)

Ambiguity is described by families of transition matrices, $(\mathcal{U}^{(m)})_{m\in\mathcal{M}}$ for the dynamics of $X^{(m)}$, which may vary in time.

The construction is then based on Whittle indexibility [77]. In particular, they reduce the problem to considering two bandits, where one bandit always generates a constant cost $\gamma$ and the other bandit is identical to the $m$th bandit. The worst-case expected cost obtained when starting in state $i$ in the $m$th bandit, $V^{(m)}(i)$, allowing any combination of transition rates, will then satisfy the robust dynamic programming principle, that is,

[TABLE]

Let $D^{(m)}(\gamma)\subseteq\mathcal{X}^{(m)}$ be the set of states for which it is optimal to rest the $m$th bandit when the reward of the constant bandit is $\gamma$. Caro and Gupta show that the robust bandit is Whittle indexible in the sense that $D^{(m)}(\gamma)$ increases monotonically from $\phi$ to $\mathcal{X}^{(m)}$ as $\gamma$ increases from $-\infty$ to $+\infty$. The index of the $m$th bandit at state $i$ is the unique value $\gamma$ such that the player is indifferent between playing the $m$th bandit and the constant bandit.

This index can be characterized by

[TABLE]

where $\mathcal{Q}^{(m)}$ is the family of measures corresponding to the family of transition matrices $\mathcal{U}^{(m)}$.

Unfortunately, as discussed in Caro and Gupta [20], the robust Gittins index (2.4) does not yield a strategy optimizing the robust Bellman equation

[TABLE]

where $(i_{1},...,i_{K})\in\prod_{m=1}^{M}\mathcal{X}^{(m)}$.

In short, this non-optimality arises due to the fact that the robust Bellman equation introduces dependency between bandits. In particular, at equilibrium, the adversary (who determines the transition probabilities for each bandit) may choose differently depending on the state of all bandits, rather than just the bandit of interest.

The index (2.4) also can be interpreted as a Lagrangian relaxation of the optimal control problem (see also Gocgun and Ghate [43]). The natural question that arises is, ‘Does this relaxation satisfy some adjusted notion of optimality?’

In this paper, we propose a new form of optimality in terms of compensators of the value function. This can be seen as a relaxation of the dynamic programming principle through the martingale optimality principle, in order to address a control problem under an inconsistent nonlinear operator. We will show that the strategy given by robust Gittins index satisfies this optimality criteria. We also allow the cost to be continuous valued and non-Markovian as in El Karoui and Karatzas [32]. This allows the study of various numerical methods to estimate our probabilistic state in the learning problem, whereas the numerical method in Caro and Gupta [20] is limited to finite state Markov process. A simple numerical example then allows us to observe some qualitative peculiarities given the interaction between uncertainty aversion and learning.

Remark 5*.*

In a non-Markovian framework, Whittle indexibility is not well-defined. Hence, the interpretation of optimality is required to understand a solution to the multi-armed bandit problem under uncertainty aversion.

Remark 6*.*

Li [56] considers a Bayesian formulation for the index but allowing for multiple priors. The focus is on describing how the set of uncertainty affects the index, but without proving any form of Whittle indexibility. Our models also verify and generalize these results.

3 Uncertainty, Nonlinear Expectation and Optimality

In this section, we will outline how ‘nonlinear expectation’ operators can be used to model Knightian uncertainty. We will also discuss how we can use these tools to study a control problem, under uncertainty, while retaining some form of time consistency. We will build on the modelling framework of El Karoui and Karatzas [32] as proposed in Assumption 2.1.

We will first outline our setup and the additional assumptions we use in our study of the robust bandit problem. We will use a ‘nonlinear expectation’ $\mathcal{E}^{(m)}$ (Assumption 3.6) to model uncertainty on the space \big{(}\Omega^{(m)},\mathbb{P}^{(m)},(\mathcal{F}^{(m)}_{t})_{t\geq 0}\big{)} of a single bandit, and then extend our uncertainty to the orthant joint space (Definition 2.3) via the combined nonlinear expectation $\mathfrak{E}$ (Definition 3.7). We will omit the superscript $(m)$ when it is clear from context.

In order to avoid technical difficulties, we will make the following assumption on the cost processes.

Assumption 3.1**.**

For each $m\in\mathcal{M}$, there exists $C^{(m)}<\infty$ such that

[TABLE]

Assumption 3.1 is purely technical. We may replace boundedness of $h^{(m)}$ by an integrability assumption on the total discounted cost (as in [32]); we then need to generalize the domain of the nonlinear expectation. We can also remove the assumption on the convergence of $h^{(m)}$ to its bound, but we then need to take more care to ensure that the stopping times we considered in (2.4) and elsewhere can be assumed to be a.s. finite. Given the discount factor, this assumption does not have large impact on our modelling.

3.1 Nonlinear Expectations and Time Consistency

We now focus on the filtered probability space $\left(\Omega,\mathbb{P},\left(\mathcal{F}_{t}\right)_{t\geq 0}\right)$ modelling the returns from playing a single bandit. As in Peng [61], we define a nonlinear expectation as follows:

Definition 3.2**.**

A system of operators

[TABLE]

for $t\in\mathbb{T}:=\{0,1,2,...\}$ is said to be an $\left(\mathcal{F}_{t}\right)_{t\geq 0}$-consistent coherent nonlinear expectation if it satisfies the following properties: for $X_{n},X,Y\in L^{\infty}(\mathcal{F}_{\infty})$ and $c\in L^{\infty}(\mathcal{F}_{t})$, with all (in)equalities holding $\mathbb{P}$-a.s, we have

1. (i)

Strict Monotonicity: If $X\geq Y$ then $\mathcal{E}(X|\mathcal{F}_{t})\geq\mathcal{E}(Y|\mathcal{F}_{t})$ If, in addition, $\mathcal{E}(X|\mathcal{F}_{t})=\mathcal{E}(Y|\mathcal{F}_{t})$, then $X=Y$. 2. (ii)

$(\mathcal{F}_{t})_{t\geq 0}$-Translation Equivariance: $\mathcal{E}(X+c|\mathcal{F}_{t})=\mathcal{E}(X|\mathcal{F}_{t})+c$. 3. (iii)

Subadditivity: $\mathcal{E}(X+Y|\mathcal{F}_{t})\leq\mathcal{E}(X|\mathcal{F}_{t})+\mathcal{E}(Y|\mathcal{F}_{t})$. 4. (iv)

$\left(\mathcal{F}_{t}\right)_{t\geq 0}$-Positive Homogeneity: $\mathcal{E}(cX|\mathcal{F}_{t})=c\;\mathcal{E}(X|\mathcal{F}_{t})$ if $c\geq 0$. 5. (v)

Lebesgue property: If $\{X_{n}\}_{n\in\mathbb{N}}$ is uniformly $\mathbb{P}$-a.s. bounded and $X_{n}\to X$ $\mathbb{P}$-a.s. then $\mathcal{E}(X_{n}|\mathcal{F}_{t})\to\mathcal{E}(X|\mathcal{F}_{t})$ $\mathbb{P}$-a.s. 6. (vi)

$\left(\mathcal{F}_{t}\right)_{t\geq 0}$-consistency: for $0\leq s\leq t$, \mathcal{E}(X|\mathcal{F}_{s})=\mathcal{E}\big{(}\mathcal{E}(X|\mathcal{F}_{t})\big{|}\mathcal{F}_{s}\big{)}.

We write $\mathcal{E}(\;\cdot\;)$ for \mathcal{E}\big{(}\cdot\big{|}\mathcal{F}_{0}\big{)}.

Remark 7*.*

For simplicity, we assume the Lebesgue property throughout this paper. In the static case, upper semi-continuity can be shown to be equivalent to the Lebesgue property over $L^{\infty}$ (see [36, Corollary 4.38]). Moreover, if the operator $\mathcal{E}$ is induced by a BSDE (as in [27, 28, 61, 34] and many other papers), then the Lebesgue property typically follows from the $L^{2}$-continuous dependence of the BSDE on its terminal value.

Remark 8*.*

It is also known (see e.g. Detlefsen and Scandolo [30]) that any coherent nonlinear expectation satisfies the $(\mathcal{F}_{t})$-regularity property. That is, for any $X,Y\in L^{\infty}(\mathcal{F}_{\infty})$ and $A\in\mathcal{F}_{t}$,

[TABLE]

In particular, \mathcal{E}\big{(}X\mathbb{I}_{A}\big{|}\mathcal{F}_{t}\big{)}=\mathbb{I}_{A}\mathcal{E}\big{(}X\big{|}\mathcal{F}_{t}\big{)}.

In order to study decision making, we often require a conditional expectation defined at a stopping time. As we are working in discrete time, this is an easy construction.

Definition 3.3**.**

Given a consistent coherent nonlinear expectation $\mathcal{E}$ and a stopping time $\tau\leq T$, we define the conditional expectation at $\tau$ by

[TABLE]

With this definition, the following easy observations can be made.

Proposition 3.4**.**

The operator \mathcal{E}\big{(}\cdot\big{|}\mathcal{F}_{\tau}\big{)} satisfies the conditions of Definition 3.2 with $s$ and $t$ are replaced by stopping times.

Nonlinear expectations are well suited to the study of Knightian uncertainty, that is, uncertainty over the probability measure. This is most easily seen through the robust representation theorem (over a finite horizon) given by Artzner et al. [5], see also Föllmer and Schied [36] and Frittelli and Rosazza-Gianin [37]. Extensions to a dynamic setting are also considered by Detlefsen and Scandolo [30], Föllmer and Schied [36] and Riedel [67]. We state a version of this result which is dynamic over stopping times.

Theorem 3.5**.**

Let $\mathcal{E}$ be a consistent coherent nonlinear expectation. If there exists $T<\infty$ such that $\mathcal{F}_{\infty}=\mathcal{F}_{T}$, then $\mathcal{E}$ admits the representation

[TABLE]

where $\tau$ is a stopping time and ${\mathcal{Q}}\subseteq\left\{\mathbb{Q}:\mathbb{Q}\approx\mathbb{P}\right\}$, and the essential supremum is taken in $L^{\infty}(\mathcal{F}_{\tau},\mathbb{P})$.

Proof.

See Föllmer and Schied [36, Theorem 11.22] (with further discussion in Föllmer and Penner [35]). The sensitivity assumption assumed in these references (i.e. for every nonnegative nonconstant $X\in L^{\infty}(\mathcal{F}_{T})$, there exists $\lambda>0$ such that \mathcal{E}\big{(}\lambda X\big{)}>0) follows from strict monotonicity in our definition. ∎

Remark 9*.*

Theorem 3.5 can be obtained by construction via considering the stability of the pasting in the family $\mathcal{Q}$. (See e.g. Bion-Nadal [15] and Artzner et al. [6]).

3.2 Uncertainty on multiple bandits

In the classical Gittins theorem, independence is crucial to separate the behaviour of different bandits. In the robust representation (Theorem 3.5) we have seen that a nonlinear expectation can be viewed as the supremum of classical expectations over a family of probability measures. Therefore, the notion of independence between bandits becomes ambiguous, as statistical independence is based on the probability measure. Thanks to our explicit construction of the space (Definition 2.3), we can explicitly construct a nonlinear expectation space where each bandit remains independent.

Remark 10*.*

In [63], Peng proposed a definition of independence for a nonlinear expectation. In his approach, independence is not a symmetric relation, but typically describes independence based on the order of events: often ‘$Y$ is independent of $X$’ when $Y$ occurs after $X$. In the setting of multiple bandits, the order of events cannot be pre-identified, as it depends on the control chosen. Hence, it is not clear how to exploit the independence notion of [63] in this setting.

Let us make the last universal assumption in our paper, which describes model uncertainty for each individual bandit in our problem inspired by the robust representation (Theorem 3.5).

Assumption 3.6**.**

For each $m\in\mathcal{M}$, we have a $(\mathcal{F}^{(m)}_{t})_{t\geq 0}$-consistent coherent nonlinear expectation, $(\mathcal{E}^{(m)}(\;\cdot\;|\mathcal{F}_{\tau}))_{\tau\in\mathcal{T}^{(m)}}$ defined on the space $L^{\infty}(\Omega^{(m)},\mathbb{P}^{(m)})$ which admits the representation

[TABLE]

whenever ${S}^{(m)}$ is an $(\mathcal{F}^{(m)}_{t})$-stopping time.

Definition 3.7**.**

We define the partially consistent orthant nonlinear expectation $\left(\mathfrak{E}_{S}\right)_{S\in\mathfrak{T}(\mathcal{S})}$, to be the family of operators

[TABLE]

where, with $\mathcal{Q}^{(m)}$ as in Assumption 3.6,

[TABLE]

We also write $\mathfrak{E}$ for $\mathfrak{E}_{0}$.

Remark 11*.*

As $\bar{\mathbb{P}}=\bigotimes_{m=1}^{M}\mathbb{P}^{(m)}$ is a dominating measure for $\mathcal{Q}$, we easily observe that if $X=Y$ $\bar{\mathbb{P}}$-a.s., then $\mathfrak{E}_{S}(X)=\mathfrak{E}_{S}(Y)$ $\bar{\mathbb{P}}$-a.s. for all $S\in\mathfrak{T}(\mathcal{S})$.

Proposition 3.8**.**

The system of operators $(\mathfrak{E}_{S})$ satisfies the following properties.

1. (i)

The properties (i)-(v) in Definition 3.2 (with appropriate replacements on the operator and $\sigma$-algebra) hold for the operator $\mathfrak{E}_{S}$ $\;:S\in\mathfrak{T}(\mathcal{S})$ (i.e. strict monotonicity, translation equivariance, subadditivity, positive homogeneity and the Lebesgue property hold for $\mathfrak{E}$). 2. (ii)

Sub-consistency: For $S,S^{\prime}\in\mathfrak{T}(\mathcal{S})$ with $S\leq S^{\prime}$, we have

[TABLE]

In particular, for any measurable $X$, if $\mathfrak{E}_{S^{\prime}}(X)\leq 0$ $\bar{\mathbb{P}}$-a.s., then for any $A\in\mathcal{F}_{S^{\prime}}$ we have $\mathfrak{E}_{S}(\mathbb{I}_{A}X)\leq 0$ $\bar{\mathbb{P}}$-a.s. 3. (iii)

Independence: Let $Y$ be a random variable on $(\bar{\Omega},\mathcal{F}(\infty))$ given by

[TABLE]

where, for each $m\in\mathcal{M}$, we have a non-negative random variable $X^{(m)}$ defined on $(\Omega^{(m)},\mathcal{F}^{(m)}_{\infty})$. Then

[TABLE] 4. (iv)

Marginal projection: For a given $m\in\mathcal{M}$, let $X$ be a random variable defined on $(\Omega^{(m)},\mathcal{F}^{(m)}_{\infty})$. Define $\tilde{X}:\bar{\Omega}\to\mathbb{R}$ by $\tilde{X}(\omega^{(1)},...,\omega^{(M)})=X(\omega^{(m)})$. We then have

[TABLE]

Proof.

See Proposition C.4 in the appendix . ∎

Remark 12*.*

We deliberately choose our nonlinear expectation $\mathfrak{E}$ to be defined on a product space to simplify our discussion on the existence of the operator. In fact, one can simply weaken our assumption by having a nonlinear expectation $\mathfrak{E}$ on a joint filtration (as in El Karoui and Karatzas [32]) such that the Proposition 3.8 holds. All proofs are identical except the proof of Theorem B.3 (in the Appendix). We just need an extra step to show that the product of the marginal probability measure is also a probability measure considered under the robust representation of $\mathfrak{E}$.

In the proposition above, we have seen that $\mathfrak{E}$ is sub-consistent on the orthant filtration. However, $\mathfrak{E}$ is not consistent in the sense of Definition 3.2, i.e. if $S\leq S^{\prime}$ (componentwise), it is not necessarily the case that $\mathfrak{E}_{S}(\;\cdot\;)=\mathfrak{E}_{S}\left(\mathfrak{E}_{S^{\prime}}(\;\cdot\;)\right)$. A counterexample can be easily constructed based on the following:

Example 3.9**.**

Let $X$ and $\tilde{X}$ be random variables taking values in $\{0,1\}$ and defined on different spaces $\Omega$ and $\tilde{\Omega}$. Let $\mathcal{Q}$ and $\tilde{\mathcal{Q}}$ be families of probability measures defined on these spaces. Suppose that for all $p\in[0,1]$ there exists $\mathbb{Q}\in\mathcal{Q}$ such that $\mathbb{Q}(X=0)=p$ and that for all $\tilde{\mathbb{Q}}\in\tilde{\mathcal{Q}}$, $\tilde{\mathbb{Q}}(\tilde{X}=0)=1/2.$ Let $f:\mathbb{R}^{2}\to\mathbb{R}$ be a given function. Then it is easy to show that

[TABLE]

but

[TABLE]

By considering $\mathcal{F}_{1}^{(1)}=\sigma(X)$ and $\mathcal{F}_{1}^{(2)}=\sigma(\tilde{X})$, and defining nonlinear expectation using supremum over the family $\mathcal{Q}$ and $\tilde{\mathcal{Q}}$, the above result shows that the joint operator $\mathfrak{E}$ is not consistent. In particular, we can find a function $f$ such that

[TABLE]

3.3 Optimality

We have discussed in the previous section that the robust Gittins index (2.4) in the sense of Caro and Gupta [20] is not optimal, as it does not lead to a solution of the robust Bellman equation (discussed in [46, 58]). In order to understand what sense of optimality the robust index strategy does satisfy, we will first consider a form of optimality criteria used by El Karoui and Karatzas [32].

Let us consider an abstract stochastic control problem on a space $(\bar{\Omega},\bar{\mathcal{F}},\bar{\mathbb{P}})$ in which a choice of control $\rho$ results in an instantaneous cost process \big{(}g^{\rho}(n)\big{)}_{n\geq 1}. We may view $g^{\rho}(n)$ as a cost occured at time $n$. For example, we have $g^{\rho}(n):=\beta^{n}h^{(\rho_{n-1})}(t^{\rho}_{n})$ in (2.2). We can also define the filtration of information obtained up to time $n$ when following $\rho$ by

[TABLE]

where $\tilde{\eta}$ is the corresponding Mandelbaum allocation sequence (Definition 2.4, Remark 2). We will discuss this filtration in detail in Remarks 22 and 23.

Remark 13*.*

It is clear from the definition that the strategy process $(\rho_{n})_{n\geq 0}$ is $(\mathcal{G}^{\rho}_{n})_{n\geq 0}$-adapted. We will show later that the cost process $g^{\rho}(n):=\beta^{n}h^{(\rho_{n-1})}(t^{\rho}_{n})$ (as in (2.2)) is also adapted with respect to $\mathcal{G}^{\rho}_{n}$.

Suppose that we are given a nonlinear expectation operator $\mathfrak{E}$, as in Definition 3.7, and consider a minimization problem over the space of Mandelbaum allocation strategies, as represented by their equivalent form $\rho$ (Remark 2). The process $\rho$ not only describes our strategy and the corresponding cost, but also determines the observed filtration. Therefore, at any point in time, it does not make sense to compare strategies unless those strategies yield the same information at the considered time.

Definition 3.10**.**

We say strategies $\rho$ and $\rho^{\prime}$ are historically equivalent at time $N$, denoted by $\rho\sim_{N}\rho^{\prime}$, if $\rho_{n}=\rho^{\prime}_{n}$ for all $n\leq N$.

Remark 14*.*

For every strategy $\rho$, we have $\rho\sim_{0}\rho^{*}$.

We can now give a standard form of optimality which is often considered when we have a consistent nonlinear expectation operator.

Definition 3.11**.**

We say a strategy $\rho^{*}$ is a strong optimum if for every strategy $\rho$ such that $\rho\sim_{N}\rho^{*}$, we have

[TABLE]

Remark 15*.*

When $\mathfrak{E}$ is replaced by an $(\mathcal{F}_{n})$-consistent nonlinear expectation and $(\mathcal{G}^{\rho}_{N})$ is replaced by $(\mathcal{F}_{N})$, strong optimality simplifies to

[TABLE]

A standard approach to tackle the decision making under time-inconsistency (nonlinear expectation) operator is to define ‘the optimal strategy’ through the solution of the robust Bellman equation [46, 58] as considered in Caro and Gupta [20]. Using the tower property, we can show that the strong optimum under $(\mathcal{F}_{n})$-consistent nonlinear expectation is equivalent to the solution to the robust Bellman equation.

3.4 C-Optimality

In the bandit setting, our nonlinear expectation is not necessary (time-)consistent. In order to understand the Gittins index strategy under an inconsistent operator, we propose an alternative notion of optimality, which is inspired by martingale optimality.

For motivation, consider an $(\mathcal{F}_{n})$-consistent nonlinear expectation $\mathcal{E}$. Suppose that we wish to solve the minimization problem

[TABLE]

For a given strategy $\rho$, we define a process $X^{\rho}_{N}:=\sum_{n=1}^{N}g^{\rho}(n)+V_{N}.$ Under mild conditions, we know from the martingale optimality principle that $(X^{\rho}_{N})$ is an $\mathcal{E}$-submartingale for every strategy $\rho$ and it is a martingale for an optimal strategy $\rho^{*}$.

By using the Doob–Meyer decomposition for nonlinear expectation (see e.g. [26, Theorem 8]), we can write

[TABLE]

where $(M^{\rho}_{N})$ is an $\mathcal{E}$-martingale and $(C^{\rho}(n))$ is a non-negative predictable process with $C^{\rho}(n)\equiv 0$ for the optimal strategy $\rho^{*}$.

By rearranging the equation above, for every $\rho$,

[TABLE]

Moreover, for an optimal strategy $\rho^{*}$, we have

[TABLE]

Inspired by the analysis above, we propose an alternative notion of optimality in an inconsistent setting.

Definition 3.12**.**

We say a strategy $\rho^{*}$ is C-optimal if there exists a $(\mathcal{G}^{\rho^{*}}_{n})$-adapted process $(V_{n})$ (called a value process) and a collection of random variables $(C_{N}^{\rho}(n))_{N,n\geq N+1,\rho\sim_{N}\rho^{*}}$ (called a (sub-)compensator) such that

1. (i)

$n\mapsto C_{N}^{\rho}(n)$ is a $(\mathcal{G}^{\rho}_{n})$-predictable process, 2. (ii)

$N\mapsto C_{N}^{\rho}(n)$ is non-increasing, 3. (iii)

For every strategy $\rho\sim_{N}\rho^{*}$,

[TABLE]

with equality for $\rho=\rho^{*}$, 4. (iv)

For every strategy $\rho\sim_{N}\rho^{*}$,

[TABLE]

We can see \big{(}C_{N}^{\rho}(n)\big{)} acts as ‘(sub-)compensator’ to the cost, and $V_{N}$ acts as the value function. This approach is loosely related to the capital requirement approach discussed by Frittelli and Scandolo [38]. We can interpret Definition 3.12 as requiring that the (sub-)compensators $(C_{N}^{\rho}(n))$

1. (i)

is known one-step in advanced before observing the cost (i). 2. (ii)

consistently (sub-)compensate the cost. In particular, as time elapses, we obtain more information and thus require the same amount, or possibly less to (sub-)compensate (ii). 3. (iii)

complement the extra cost occurred for a sub-optimal strategy (iii). 4. (iv)

are bounded below by a compensator of a particular strategy $\rho^{*}$, which we call ‘optimal’ (iv).

Remark 16*.*

We have mentioned the robust Bellman equation [46, 58] as an approach to force time-consistency in our decision making. The fundamental idea of this approach is to freeze our value function and propagate its value backward in time. In particular, suppose we have $V^{*}_{n+1}$ as our expected remaining cost at time $n$. We then define an optimal strategy at time $n$ to be a strategy $\rho_{n}$ such that $g^{\rho}(n)+V^{*}_{n+1}$ is optimized.

A closely related approach to ensure time-consistency was proposed by Strotz [73] and Pollak [64] and developed further in Peleg and Yaari [60] and Koopmans [53]. Recent extensions include Björk and Murgoci [17], Björk, Khapko and Murgoci [16], Yong [79] and Hu, Jin and Zhou [44]. For a problem with horizon $L$, suppose that the optimal control is determined after time $n$, in other words, $(\rho^{*}_{n+1},...,\rho^{*}_{L})$ is known. We then find a control $\rho^{*}_{n}$ at time $n$ to optimize over the space of possible strategies $\left\{\rho\;\;:\;\;(\rho_{n+1},...,\rho_{L})=(\rho^{*}_{n+1},...,\rho^{*}_{L})\right\}$. In this way, the (optimal) control rather than the value function, is constructed recursively. This idea is then extended by searching for (sub-game perfect) Nash equilibria, to allow for non-uniqueness of the optimal controls.

As discussed in Section 2.1.2, the robust Bellman approach may introduce some dependency between bandits in our system. Hence, Gittins index strategy is not optimal under that approach. On the other hand, when considering a system of bandits, the measurability of our future states are determined by our current action. Therefore, the $\sigma$-algebra that is used to define the future control $(\rho^{*}_{k})_{k\geq n+1}$ cannot be chosen independently of our current control. This means that we cannot directly consider the Strotz–Pollak approach for the bandit setting as we cannot freeze our future control without freezing our current control.

The notion of C-optimality can be loosely interpreted as a third variation on these time-consistency approaches. In particular, we can interpret the compensator process as propagating a value backward in time, as in the robust Bellman approach. Optimality can then be defined forward in time, which relaxes the dependence on the filtration.

3.5 Endowment Effect

One natural question to ask is whether we can give an interpretation of C-optimality (Definition 3.12) in terms of classical strong optimality (Definition 3.11). To see this, we will consider an endowment effect through the strong optimality.

Example 3.13**.**

Let $H$ and $G$ be random variables representing the cost of two strategies and $\mathcal{Q}$ be a family of probability measures such that $H$ and $G$ are independent under each $\mathbb{Q}\in\mathcal{Q}$. Suppose $\{\mathbb{E}^{\mathbb{Q}}(H)\}_{\mathbb{Q}\in\mathcal{Q}}=[\underline{h},\bar{h}]$ and similarly for $G$. Suppose further that $\bar{h}<\bar{g}$ but $\bar{h}-\underline{g}>\bar{g}-\underline{h}$. Then for $\mathfrak{E}\left(\;\cdot\;\right):=\sup_{\mathbb{Q}\in\mathcal{Q}}\mathbb{E}^{\mathbb{Q}}\left(\;\cdot\;\right)$, we have

[TABLE]

From these inequalities, we see that, without any endowment, we strictly prefer $H$ to $G$ whereas our preference reverses with an endowment $(H+G)/2$. We know that in the classical linear expectation theory (where the classical Gittins theorem holds), an endowment does not affect our preference in the strategy.

In this section, we will show that C-optimality is nearly equivalent to a strong optimality ‘up to an endowment’ when our nonlinear expectation is time-consistent.

The following proposition follows from the definition of C-optimality and monotonicity of nonlinear expectation (in particular, Definition 3.11(iii)-(iv)).

Proposition 3.14**.**

Let $\rho^{*}$ be a C-optimal strategy with a predictable compensator $(C^{\rho^{*}}_{N}(n))$. Then for every $\rho\sim_{N}\rho^{*}$ and $A\in\mathcal{G}^{\rho^{*}}_{N}$,

[TABLE]

Let consider the case when $\mathcal{G}^{\rho}_{N}=\mathcal{F}_{N}$ for every strategy $\rho$ and pretend that $\mathfrak{E}$ is an $(\mathcal{F}_{n})$-consistent nonlinear expectation operator. Then (3.5) says that C-optimality implies strong optimality, when our agent is given the predictable endowment $-\sum_{n=N+1}^{\infty}C^{\rho^{*}}_{N}(n)$ at time $N$. We will now show that a converse result also holds, when our operator is consistent.

Definition 3.15**.**

Let $\mathcal{E}$ be an $(\mathcal{F}_{n})$-consistent nonlinear expectation. We say a strategy $\rho^{*}$ is optimal up to a predictable endowment if there exists a family of random variables $(D_{N}(n))$ such that

1. (i)

$n\mapsto D_{N}(n)$ is an $(\mathcal{F}_{n})$-predictable process, 2. (ii)

$N\mapsto D_{N}(n)$ is non-increasing, 3. (iii)

For every strategy $\rho\sim_{N}\rho^{*}$, for all $A\in\mathcal{G}^{\rho}_{N}$,

[TABLE]

or equivalently,

[TABLE]

Proposition 3.16**.**

Suppose that $\rho^{*}$ is an optimal strategy up to a predictable endowment, then $\rho^{*}$ is C-optimal.

Proof.

Take $C^{\rho}_{N}(n)=D_{N}(n)$ and V_{N}=\mathcal{E}\big{(}\sum_{n=N+1}^{\infty}\big{(}g^{\rho^{*}}(n)-D_{N}(n)\big{)}\big{|}\mathcal{F}_{N}\big{)}. ∎

In the coming section, we will show that Gittins theorem holds in the sense of guaranteeing C-optimality under an operator $\mathfrak{E}$. This means that we prove that Gittins theorem is a (strong) optimum up to some predictable endowment.

Remark 17*.*

It is an open question under which conditions the C-optimum is unique. In the most trivial case when our operator $\mathfrak{E}$ is simply a classical expectation, the endowment never affects our evaluation; thus it is reduced to the uniqueness of the value function in the classical setting.

4 Overview of Bandits under uncertainty

Let us recall that the objective of our problem is to dynamically allocate a single resource amongst $M$ bandits to minimize the total discounted cost. We have made a few assumptions to model uncertainty in the cost process which can be founded in Assumptions 2.1, 3.1 and 3.6.

We also introduce a Mandelbaum allocation strategy (Definition 2.4) and the equivalent notion $\rho$ (Remark 2) representing the choice of our control. We are now ready to establish a robust Gittins theorem with optimality in the sense of Definition 3.12. Our robust Gittins theorem generalize the result of El Karoui and Karatzas [32] to the uncertain case. One may also see this result as providing a sense of optimality for the index strategy considered by Caro and Gupta [20] and Li [56].

4.1 Robust Gittins theorem

We will first give an alternative definition to the robust Gittins’ index inspired by Weber [75], which is more convenient to use in our analysis.

Definition 4.1**.**

For each $s\geq 0$, we define the robust Gittins index of the $m$th bandit by

[TABLE]

where $\mathcal{T}^{(m)}(s)$ is the space of positive $(\mathcal{F}^{(m)}_{s+t})_{t\geq 0}$-stopping times777Equivalently, for $\tau\in\mathcal{T}(s)$, $s+\tau$ is an $(\mathcal{F}_{t})_{t\geq 0}$-stopping time. and the outer essential infimum is taken in $L^{\infty}(\mathcal{F}_{s}^{(m)})$.

By using the results proved in the later sections, we can write the robust Gittins index explicitly. We present this result here for clarity, but make no use of it in subsequent arguments.

Theorem 4.2**.**

Let $\gamma(s)$ be the robust Gittins index (Definition 4.1) (with superscript $(m)$ omitted). Then

[TABLE]

where $\mathcal{Q}$ is the family of probability measures defined in Theorem 3.5.

Proof.

See Theorem C.3 in the appendix. ∎

Recall that $\mathfrak{E}$ is the partially consistent orthant nonlinear expectation induced by the family $\left(\mathcal{E}^{(m)}\right)_{m\in\mathcal{M}}$ as given in Definition 3.7. We can obtain an optimal allocation strategy by considering the following theorem.

Theorem 4.3** (Robust Gittins theorem).**

Suppose that for each $m\in\mathcal{M}$, $(\mathcal{F}^{(m)}_{t})_{t\geq 0}$ is generated by some underlying process $(\xi^{(m)}_{t})_{t\geq 1}$. Let $\psi^{(m)}_{n}$ be the total number of trials of the $m$th bandit before the $n$th play of the system. i.e. $\psi^{(m)}_{n}:=\sum_{k=0}^{n-1}\mathbb{I}(\rho^{*}_{k}=m)$ (given an allocation strategy $\rho^{*}$ up to time $n-1$).

Then the allocation strategy $\rho^{*}$ given (recursively) by

[TABLE]

is C-optimal (Definition 3.12) under $\mathfrak{E}$ for the cost

[TABLE]

Remark 18*.*

We choose $\rho^{*}$ to be the minimum value in the (random) set of minimum Gittins index machines $\{\operatorname*{arg\,min}_{k}\gamma^{(k)}(\psi^{(k)}_{n})\}$ as a simple method of symmetry breaking, in order to avoid complexities due to measurable selection. In fact, any choice of $\rho^{*}_{n}\in\{\operatorname*{arg\,min}_{k}\gamma^{(k)}(\psi^{(k)}_{n})\}$ also yields C-optimality.

Remark 19*.*

The robust Gittins theorem states that an optimal choice is given by always playing a bandit with the lowest robust Gittins index. At each time, the indices of unplayed bandits do not change. This leads to a form of consistency in the values associated with different bandits, even though $\mathfrak{E}$ is not consistent.

4.2 Sketch of the Proof

We will separate the proof into two parts: In Part A, we analyze a one-armed bandit in a robust setting. In Part B, we combine $M$ bandits together. The main body of the rigorous proof can be found in Appendices A and B (respectively) as self-explained sections. We summarize the structure and approach of the proof here.

4.2.1 One-armed bandit optimality

We begin by considering play of the $m$th machine (with the superscript $(m)$ omitted).

Step A.1

Observe that the robust Gittins index is the minimum compensation for which we are willing to continue to play the bandit (with compensation).

By minimality, the net expected cost under optimal play must be zero (Theorem A.2), i.e.

[TABLE]

In particular, for any subsequent stopping time $\tau\in\mathcal{T}(s)$, we have

[TABLE]

Step A.2

We view the process $\gamma$ as the ‘average’ cost of playing the bandit. Once the process $(\gamma(t))_{t\geq s}$ exceeds $\gamma(s)$, the reward $\gamma(s)$ will no longer be sufficient to encourage continued play; so it will be optimal to stop. In particular, the stopping time

[TABLE]

yields equality in (4.2) (Theorem A.8).

Step A.3

Imagine that, whenever the bandit (with compensating reward) is no longer attractive to play, we were to increase the compensation sufficiently to make ourselves indifferent to continuing. The expected value of future loss, with this increased compensation, must again be zero (Proposition A.11). The offered compensation can be written as a running maximum of the robust Gittins index process and we can express the expected return

[TABLE]

With the compensation reward $\left(\Gamma(t)\right)$, we are always willing to continue to play. In particular, at any point in time, we have a non-positive expected future cost (Theorem A.13), i.e.

[TABLE]

Step A.4

Now suppose we were to take a break from playing for some period, and then resume our earlier strategy. In this case, we may lose some expected profit (Equation (4.5)) due to the discount effect of the delay. .

By (4.4), the total reward of this game is zero. Therefore, the delay of getting the reward must result in a possibly worse outcome. In Theorem A.14, we use this observation, together with the robust representation (Assumption 3.6) to show that for any fixed $\epsilon>0$ there is a probability measure $\mathbb{Q}\in\mathcal{Q}$ such that, for every decreasing predictable process $(\alpha(t))$ taking values in $[0,1]$,

[TABLE]

Remark 20*.*

Step A.4 is the key point in which positive homogeneity of $\mathcal{E}$ is used. A predictable process $(\alpha(t))$ represents the delay due to taking a break to play another bandit. In step A.3, we choose the compensator such that the total expected return is zero but the bandit is always attractive to be played. (i.e. we always have a reward for the future.) We therefore cannot expect a better outcome than zero if we delay our play. Mathematically, one can replace positive homogeneity and subadditivity by convexity and the property that: if $\mathcal{E}(X|\mathcal{F}_{t})\leq 0$, then for all $\mathcal{F}_{t}$-measurable random variables $\alpha$ taking values in $[0,1]$, we have $\mathcal{E}(\alpha X|\mathcal{F}_{t})\geq\mathcal{E}(X|\mathcal{F}_{t})$.

4.2.2 Information structures for Multi-armed bandits

We now consider combining play over multiple machines.

To retain consistency for a single bandit, the nonlinear expectation needs to be defined together with the filtration. It follows that we need to define an ‘independent’ nonlinear expectation on the joint space of the bandits, which we do via an orthogonal product space. This restriction does not allow us to directly implement Mandelbaum’s [57] original approach for a dynamic allocation strategy (Definition 2.4). This is because the multi-parameter process $(\tilde{\eta}(n))$ is only defined to be measurable with respect to the orthant filtration. In particular, it is not clear how one could directly extract the component of $(\tilde{\eta}(n))$ to the marginal space $\Omega^{(m)}$ where our single-bandit nonlinear expectation is defined.

The importance of decomposing a strategy on the multi-armed bandit to strategies for one-armed bandits can be seen in the proof of El Karoui and Karatzas [32, Equation 5.1] (via Whittle’s approach [76]), and is described more explicitly in their continuous time paper [33, Equation 6.9].

In order to overcome this difficulty, we introduce a class of allocation strategies where there is a component associated to the stopping times of the marginal filtrations. This component allows us to connect and separate the space of multiple bandits to the marginal space of each single bandit.

Our class of allocation strategies consists of two components $(\tau,p)$. The collection of random times $\tau=(\tau^{(m)}_{k})_{k\geq 0,m\in\{1,...,M\}}$ will identify the duration for which will play the $m$th bandit, the $k$th time we start to play. This sequence is chosen based on historical observations of the $m$th bandit only, that is, the random times $\sum_{k=0}^{K}\tau^{(m)}_{k}$ are $(\mathcal{F}^{(m)}_{t})_{t\geq 0}$-stopping times for all $K\geq 0$. Once we play a bandit for $\tau^{(m)}_{k}$ trials, we will then reconsider which bandit to play. Our choice of new bandit (which may be the same as before) will be described by the sequence $(p_{n})$ taking values in $\{1,...,M\}$, and may depend on information from all bandits. The allocation strategy can be defined formally as follows:

Definition 4.4**.**

We say \tau:=\big{(}\tau^{(m)}_{k}\big{)}_{k\geq 0,m\in\mathcal{M}} is a family of time allocation sequences if

1. (i)

For each $m$, $(\tau^{(m)}_{k})_{k\geq 0}$ is a sequence of non-negative random times defined on the space $(\Omega^{(m)},\mathcal{F}^{(m)}_{\infty})$. 2. (ii)

$\sum_{i=0}^{k}\tau^{(m)}_{i}$ is an $(\mathcal{F}^{(m)}_{t})_{t\geq 0}$-stopping time for all $k\geq 0$.

Intuitively, the random sequence $(p_{n})$ is allowed to depend on all prior observations from all bandits. For the sake of precise bookkeeping we need to record, at each moment, how many times we have already played each bandit. This leads to the following definition.

Definition 4.5**.**

Given a family of time allocation sequences $\tau$, we say a sequence of random variables $(\eta_{n})_{n\geq 0}$ taking values in $\mathcal{S}=\mathbb{N}_{0}^{M}$ is a recording sequence associated to $\tau$, with corresponding choice sequence $(p_{n})_{n\in\mathbb{N}_{0}}$ taking values in $\mathcal{M}$, if

1. (i)

$\eta_{0}=(0,...,0).$ 2. (ii)

$\eta_{n+1}=\eta_{n}+e^{(p_{n})}$.

The choice process $p_{n}$ satisfies

1. (iii)

for all $k\in\mathcal{M}$ and $r\in\mathcal{S}$,

[TABLE]

where $\Psi_{r}^{(m)}:=\sum_{i=0}^{r^{(m)}-1}\tau^{(m)}_{i}$. In particular, $\{\eta_{n}=r\}\in\mathcal{F}(\Psi_{r})$.

For a given time allocation sequence $\tau$, the recording sequence $\eta_{n}$ determines the decision filtration, given by

[TABLE]

where $\Psi_{r}^{(m)}=\sum_{i=0}^{r^{(m)}-1}\tau^{(m)}_{i}$.

Remark 21*.*

We can see in Definition 4.5(iii) that $(p_{n})$ is adapted to the filtration $(\mathcal{G}^{(\tau,p)}_{n})_{n\geq 0}$, i.e. we have made our decision what to do next based on our previous observations.

Definition 4.6**.**

An (admissible) allocation strategy $(\tau,p)$ consists of a family of time allocation sequences $\tau$ and a $(\mathcal{G}^{(\tau,p)}_{n})_{n\geq 0}$-adapted choice sequence $p$ (defined under $\tau$).

Example 4.7**.**

Suppose there are two bandits. The first bandit gives only 2 outcomes: $\{w,l\}$. Consider the strategy of playing the first bandit until we see the first $l$. Then we swap to the second bandit for two trials and swap back to the first bandit and repeat the same procedure.

In this case, we define $(X_{t})_{t\geq 1}$ to be the outcome of the first bandit and define $\theta_{k+1}:=\inf\{t\geq 1\;:\;X_{t+\sum_{i=0}^{k}\theta_{i}}=l\}$. We then have the representation of this strategy

[TABLE]

The corresponding recording sequence is

[TABLE]

The same strategy can be represented in multiple ways. Here, for example, we can also write

[TABLE]

The corresponding recording sequence becomes

[TABLE]

As discussed in Remark 2, we can express a Mandelbaum allocation strategy (Definition 2.4) in terms of a sequence $\rho$ of decisions made at each time. For the strategy described above, this gives the unique sequence

[TABLE]

Extending this example, we can generally write our strategy $(\tau,p)$ in terms of $\rho$ and vice versa. This unique representation provides a simple (if inefficient) description of our strategy, which we now make precise.

Definition 4.8**.**

Define the random variable $\rho_{n}$ to be the bandit which will be observed in the $n$th play under an admissible allocation strategy $(\tau,p)$. We call the process $(\rho_{n})_{n\geq 0}$, a simple form allocation sequence. The construction of the sequence $(\rho_{n})$ is given explicitly in Lemma C.8 in the appendix.

For admissible allocation strategies $(\tau,p)$ and $(\hat{\tau},\hat{p})$, we write $(\tau,p)\sim(\hat{\tau},\hat{p})$ if they lead to the same simple form. (Clearly, $\sim$ defines equivalence classes.)

Remark 22*.*

Observe that if $\rho$ is the simple form of $(\tau,p)$ and we denote the time allocation sequence \mathfrak{1}=\big{(}1,1,1,...\big{)}, then $(\mathfrak{1},\rho)$ is an allocation strategy which yields the same decisions as $(\tau,p)$. In particular, we have $(\mathfrak{1},\rho)\sim(\tau,p)$.

Furthermore, one can check that the recording sequence corresponding to $(\mathfrak{1},\rho)$ is exactly the Mandelbaum allocation strategy (Definition 2.4). In particular, we can explicitly construct a one-to-one correspondence between our equivalence class of admissible strategies (Definition 4.6) and Mandelbaum allocation strategies, and we have $\mathcal{G}^{(\mathfrak{1},\rho)}_{n}=\mathcal{G}^{\rho}_{n}$ in (3.1).

Remark 23*.*

Assume that, for $m\in\mathcal{M}$, the filtration $(\mathcal{F}^{(m)}_{t})$ is generated by an underlying real process $(\xi^{(m)}_{t})_{t\geq 1}$ defined on the space $(\Omega^{(m)},\mathcal{F}_{\infty}^{(m)})$. i.e.

[TABLE]

If we parameterize our actions by a simple form strategy $(\mathfrak{1},\rho)$ with associated recording sequence $\eta$, then $\rho_{n-1}$ is the decision made at time $n-1$ to generate the outcome observed at time $n$. The observation at the $n$th play is given by

[TABLE]

We define the observed filtration by $\mathcal{H}^{\rho}_{0}=\{\phi,\bar{\Omega}\}$ and $\mathcal{H}^{\rho}_{n}:=\sigma\left(\xi^{\rho}_{1},...,\xi^{\rho}_{n}\right)$. We prove, in the appendix, that the observed filtration agrees with that used in Definition 4.5 when considering measurability of $\rho$. That is

[TABLE]

where $\eta$ is the recording sequence corresponding to $(\mathfrak{1},\rho)$.

4.2.3 Multi-armed bandit optimality

We can now give the second half of the proof for the robust Gittins index theorem where we will consider [math] as our referencing value function.

In order to prove the optimality of the robust Gittins’ strategy, we define the target function for an allocation strategy by

[TABLE]

where $\rho$ is a simple form derived from $(\tau,p)$ and \big{(}\Gamma^{(m)}(t)\big{)} is the running max of the robust Gittins index of the $m$th bandit, as considered in (4.4).

Step B.1

Suppose that we have $M$ bandits, with associated indifference rewards $(\Gamma^{(m)})_{m\in\mathcal{M}}$ as in step A.3. If we mix the play of these bandits, this is equivalent to taking a break in a single bandit to play the others. This delay will result in a possibly worse outcome (Equation (4.6) in step A.4).

In Theorem B.3, we use the definition of $\mathfrak{E}$ and apply Fubini’s theorem to show that, for all allocation strategies $(\tau,p)$, this implies that for any $\epsilon>0$, there exists a probability measure $\bigotimes_{m=1}^{M}\mathbb{Q}^{(m)}\in\mathcal{Q}$ such that

[TABLE]

where $\tilde{\alpha}^{(m)}(t)$ is the delay effect on the $m$th bandit due to playing other bandits.

As $\epsilon$ is arbitrary, it follows that for all allocation strategies $(\tau,p)$,

[TABLE]

Step B.2

In step A.2, we noticed that the total expected loss of a single bandit between $S$ and $S^{\prime}$ is zero, for $S$ and $S^{\prime}$ the consecutive stopping times when the robust Gittins index hits a new maximum (Equation (4.3)). We use this fact to construct a family of time allocation sequences as a candidate optimal strategy.

Define (inductively) $S^{(m)}_{k}:=\sum_{l=0}^{k-1}\sigma^{(m)}_{l}$ and

[TABLE]

Using our construction on the class of allocation strategies, we can project the joint nonlinear valuation to its marginal space which is equipped with a consistent nonlinear expectation. We can then use the result from step A.2, that $\sigma^{(m)}_{k}$ yields equality in (4.2), to show that, for any $p$, with the choice of time allocation sequences $\sigma=(\sigma^{(m)}_{k})$, the allocation strategy $(\sigma,p)$ has value

[TABLE]

This result is shown in Theorem B.4.

Step B.3

By combining Step B.1 and Step B.2, for any $p$, with the choice of time allocation sequences $\sigma=(\sigma^{(m)}_{k})$ considered above, we have

[TABLE]

We consider $C^{\rho}(n)=\beta^{n}\Gamma^{(\rho_{n-1})}(t^{\rho}_{n})$ as a (sub-)compensator in Definition 3.12. The strategy $\rho^{*}$ given in Theorem 4.3 is the strategy of always playing the bandit with the minimal index. Therefore, it lies in the same equivalence class as a strategy with the time allocation sequences $(\sigma^{(m)})$ (and with $p$ indicating the minimum index amongst all bandits at each time). Hence, by (4.12),

[TABLE]

Furthermore, by (4.10),

[TABLE]

By monotonicity of the process $\Gamma$, we prefer lower value earlier, due to the discount effect. Thus, we prove the optimality condition when $N=0$. We can now restart our analysis at the considered (orthant) time to obtain the optimal condition for $N>0$. We now thus show that $\rho^{*}$ satisfies the condition for C-optimal. The formal proof of this result can be found in Theorem B.5.

5 Numerical Results

In this section, we study the behaviour of the robust Gittins index using a numerical example. Again we omit the superscript $(m)$ for notational simplicity.

We suppose the bandit under consideration generates independent identically distributed costs $(h(t))_{1\leq t\leq T}$ of either $1 or$0, given (unknown) probability $\mathbb{P}(h(t)=1)=\theta$ and $h(t)=2$ for all $t>T$. The filtration $\left(\mathcal{F}_{t}\right)_{t\geq 0}$ is generated by the observed cost process $(\xi_{t})_{1\leq t\leq T}=(h(t))_{1\leq t\leq T}$ (with $\mathcal{F}_{0}$ trivial). The horizon $T$ can be thought of as the maximum number of times that each bandit can be played.

Remark 24*.*

An imaginary horizon $T$ is introduced in order to allow us to easily construct a data-driven recursive nonlinear expectation (5.1) by backward induction

The future cost $h(t)=2$ is introduced to simplify our numerical method. By considering (4.1), we can see that the robust Gittins index $(\gamma(t))_{t\geq 1}$ takes values between [math] and $1$ when $t\leq T$ and $\gamma(t)=2$ for $t>T$. Moreover, the optimal stopping time $\sigma(t,\gamma(t))\leq T-t$. Hence, one can calculate the robust index $\gamma(t)$ by considering a finite horizon optimal stopping problem.

We model uncertainty in this setting by constructing a one-step coherent nonlinear expectation $\mathcal{E}_{(t)}(\cdot):L^{\infty}(\mathcal{F}_{t+1})\to L^{\infty}(\mathcal{F}_{t})$. Once we have a one-step coherent nonlinear expectation, we can construct an $(\mathcal{F}_{t})$-consistent coherent nonlinear expectation by

[TABLE]

Remark 25*.*

We will consider one-step coherent nonlinear expectation which is inspired by the DR-Expectation [23], see also Bielecki, Chen and Cialenco [14]):

[TABLE]

where \Theta_{t}=\Big{[}p^{-}(p_{t},n_{t}),p^{+}(p_{t},n_{t})\Big{]} corresponds to a credible interval for $\theta$ given our observations at time $t$, using a (possibly improper) Beta prior distribution. The processes $n_{t}$ and $p_{t}$ correspond to the number of observations and the (posterior mean) estimate of $\theta$ at time $t$.

In particular, we may choose a credible level $k\in[0,1]$ and obtain $p^{\pm}(p_{t},n_{t})$ by

[TABLE]

where $q\mapsto I_{(a,b)}^{-1}(q)$ is the quantile function of the $\text{Beta}(a,b)$ distribution.

One could also use the central limit theorem to obtain an asymptotic confidence interval. However, due to the fact that $\Theta_{t}\subseteq[0,1]$, we restrict ourselves to the credible set above to avoid end-effects, and allow for asymmetry in the plausible values around the ‘best’ estimate.

As our credible set is constructed from $p_{t}$ and $n_{t}$, and the pair $(p_{t},n_{t})$ can be computed recursively, it follows that for every $f:\{0,1\}^{T-t}\to\mathbb{R}$, there exists a function $g_{k,T-t}:\mathbb{R}^{2}\to\mathbb{R}$ such that \mathcal{E}\big{(}f(\xi_{t+1},...,\xi_{T})\big{|}\mathcal{F}_{t}\big{)}=g_{k,T-t}\Big{(}p_{t},\frac{1}{\sqrt{n_{t}}}\Big{)}. 888Here, we write the nonlinear expectation as a function of $n_{t}^{-1/2}$ instead of $n_{t}$ as we wish to approximate our function on a compact domain. The choice of $n_{t}^{-1/2}$ comes from the natural scaling of the credible set.

By recalling the definition of $\gamma(s)$ (Definition 4.1), one can show (using a general robust dynamic programming argument, as in Ruszczyński [71], or the nonlinear Snell’s envelope, as in Riedel [68]) that we can write

[TABLE]

for some function $\gamma_{k,\beta,T-t}$.

We then use a simple finite-difference algorithm (see Appendix D) to estimate the function

[TABLE]

where, in our simulations, we fix $n_{0}=1$.

Plots of this estimate, for various values of $k$, $\beta$ and $T$, can be found in Figure 1.

For $h(t)=\xi_{t}$, with uncertainty modeled by (5.1), at each time step we wish to play the bandit with the lowest $\theta$. Classically, this is estimated by $p$, so a naïve (greedy) strategy would suggest playing the bandit with the lowest estimated average loss $p$. By using C-optimality, at each point, we choose a bandit with the lowest $\gamma$. Therefore, we may think of $\gamma$ as an implied probability $p$, distorted to account for exploration and exploitation of the system of bandits.

In Figure 1, we see the following broad phenomena:

• •

When $1/\sqrt{n}$ is small, the difference between $\gamma$ and $p$ is close to zero. In particular, this says that when we have high certainty in our estimates, $\gamma$ is equivalent to the estimated probability.

• •

When we increase $\beta$, the difference typically $\gamma-p$ decreases. This corresponds to the fact that $\beta$ is a discount factor which determines how much we value future costs. Therefore, increasing $\beta$ increases the degree that we wish to explore the system, i.e. we become more optimistic in our evaluation. We also observe that decreasing $\beta$ also yields a similar result to shortening the horizon.

• •

When $k$ is increased, the difference $\gamma-p$ increases. This is due to the fact that $k$ corresponds to the ‘width’ of the ‘credible interval’. Hence, large $k$ means that we become more conservative and favour exploiting over exploring.

5.1 Prospect Theory

One result suggested in Figure 1 when $\beta=0.9999$ and $k=0.01$ is that, when we do not worry about uncertainty, we are more optimistic when $p$ is large (close to $1$), that is, $\gamma$ is clearly less than $p$. On the other hand, when uncertainty dominates, e.g. when $\beta=0.9999$ and $k=0.95$, or $\beta=0.95$ and $k=0.8$, we become more pessimistic.

Curiously, when $\beta=0.9999$ and $k=0.8$, or $\beta=0.95$ and $k=0.5$, both optimism and pessimism can be seen. For large $p$, (when the game seems bad), pessimism dominates, while for small $p$ (when the game seems good) we become optimistic in our optimal strategy. This gives a bias in the probabilities, related to that used in the probability weighting functions as considered in prospect theory by Kahneman and Tversky [47] or in rank-dependent expected utility by Quiggin [65, 66]. In this literature, they propose models to explain irrationality in human decisions under risk. They argue that people generally reweigh the probabilities of different outcomes using a nonlinear increasing map $p\mapsto\pi(p)$, with various assumptions on its curvature.

Our result (for appropriate values of $\beta$ and $k$) reflects this behaviour without imposing a probability weighting function as in classical prospect theory. Instead, the combination of the effect of learning and uncertainty leads to distortions of the estimated probability.

5.2 Monte-Carlo Simulation

In order to illustrate the performance of the robust Gittins index calculated above in the real decision making, we consider the Bernoulli bandit as described above over 50 exchangeable bandits and for a horizon $T=10^{4}$. We run $10^{3}$ Monte-Carlo simulations and compare performance of various strategies for decision making. To provide a wide range of scenarios in which our strategies must perform, in each simulation we first generate $a,b$ independently from a $\Gamma(1,1/100)$ distribution, then generate the ‘true’ probabilities for each bandit independently from $\text{Beta}(a,b)$. We generate $10$ trials on each bandit to provide initial information.

N.B. Formally, we assume that each bandit can be played for at least $T=10^{4}$ trials in constructing our Gittins index. We illustrate the performance of the first $10^{4}$ plays to compare with other algorithms.

5.2.1 Measures of Regret

There are a number of possible objectives to measure the loss of our decisions. We will consider the following examples (from Bertini et al. [19] and Lai and Robbins [54]).

• •

Expected–expected regret. This is the difference in the true expectations under our strategy and an optimal strategy with perfect information. In our setting, this can be given by $R(L)=\sum_{n=0}^{L}(\theta^{(\rho_{n})}-\theta^{*})$ where $\theta^{(m)}$ is the true probability of the $m$th bandit and $\theta^{*}=\min_{m}\theta^{(m)}$.

• •

Sub-optimal plays. This measures the number of times where we play a sub-optimal bandit which is given by $N_{\vee}(L)=\sum_{n=0}^{L}\mathbb{I}(\theta^{(\rho_{n})}\neq\theta^{*}).$

5.2.2 Policy for multi-armed-bandits

In our simulation, we will label our algorithm the DR (Data-Robust) algorithm. We also consider the following classical policies which are commonly used to solve the Bernoulli bandit problem. These policies choose an arm by considering the minimal index $I$ evaluated on each bandit separately. Literature about these policies and further developments can be found in the reviews by Bertini et al. [19] or Russo et al. [70]. For notational simplicity, we will denote by $p$ and $n$ the estimated probability and the number of observations of the considered bandit at the time before making a decision.

• •

Greedy strategy. In this policy, we choose the bandit with the minimal estimated probability given by $I^{Greedy}=p.$

• •

Thompson strategy. This is a Bayesian adaptive decision strategy for the bandit problem. It proceeds by first randomly generating a sample from the posterior distribution of the mean cost of each bandit, then chooses to play the bandit which gave the minimal sample. In our setting, these samples are given by $I^{Thompson}\sim\text{Beta}(a_{0}+pn,b_{0}+(1-p)n)$ where $a_{0},b_{0}>0$ are the parameters of a Beta prior distribution for the mean. To avoid biasing our estimation, we consider initial values $a_{0}=b_{0}\in\{0.0001,1,50\}$, where larger values correspond to a more informative prior.

• •

UCB strategy. This is an optimistic strategy to choose the bandit based on its lower bound. $I^{UCB}=p-\sqrt{\frac{\lambda\log{N}}{n}}$ where $\lambda>0$ is a chosen parameter, which is commonly chosen to be $2$ and $N:=\sum_{m=1}^{M}n^{(m)}$ is the total number of observations across all bandits.

Remark 26*.*

To avoid bias in the algorithms, we choose a bandit uniformly at random if there is more than one bandit with minimal index.

In Figure 2, considering first the cases where $\beta=0.9999$, we can see that an increase in the value of $k$ has a nonlinear effect on the distribution of regret. Initially, increasing $k$ appears to lead to a reduction in the typical regret, but a possible increase in the average and variability of the number of suboptimal plays. However, setting $k$ too large clearly leads to worse outcomes. This is because $k$ corresponds to the level of robustness; the more robust we are, the less willing we are to explore and the more willing we are to exploit. It follows that a large value of $k$ encourages us to exploit early, and we may not find the optimal bandit to play.

On the other hand, the discount rate $\beta$ determines how much we value our future costs. If we have a high level of robustness (large $k$) but do not value the future cost enough (small $\beta$), we may end up settling for a sub-optimal decision. This can be seen most clearly when $\beta=0.9999$ and $k=0.8$. In this case the average expected-expected regret is relatively small when compared to other strategies, but its average number of suboptimal plays is relatively high. Reducing $\beta$ to $0.95$ emphasizes these effects even further.

As discussed in the introduction, the UCB algorithm asymptotically achieves a minimal regret bound (see [54]). It does so by ensuring that, over short horizons, the algorithm explores a sufficient amount, in order to guarantee good asymptotic performance. We can see that in our simulation (with $10^{4}$ plays over $50$ bandits), the UCB algorithm is still in its high exploration regime which results in a high regret and very few optimal plays. In contrast, a Greedy algorithm always chooses an arm to play without taking into account its uncertainty (and so without considering the possibility for exploration) and therefore there is no learning in its procedure. This results in the greedy algorithm yielding a low average regret but a high average number of suboptimal plays.

5.2.3 Robustness of the DR Algorithms

In Figure 3, we illustrate the interquartile range and the standard deviation of the total expected-expected regret when $\beta=0.9999$ with different values of $k$ over $1000$ simulations. We can see that by introducing an appropriate values of $k$, we can obtain a substantial reduction in the interquartile range and the standard deviation. In particular, the DR algorithm does not only give a low average regret but also does so consistently over different simulations.

Acknowledgements: Samuel Cohen thanks the Oxford-Man Institute for research support and acknowledges the support of The Alan Turing Institute under the Engineering and Physical Sciences Research Council grant EP/N510129/1. Tanut Treetanthiploet acknowledges support of the Development and Promotion of Science and Technology Talents Project (DPST) of the Government of Thailand.

Appendix A Part A: Analysis of a single bandit

We will now flesh out the sketch given in Section 4.2.

In this section, we will focus the discussion on a single bandit.

A.1 Step A.1: Indifference reward and Optimal Stopping problem

We first recall the definition of the robust Gittins index (process).

[TABLE]

where $\mathcal{T}(s)$ denotes the family of $\left(\mathcal{F}_{s+t}\right)_{t\geq 0}$-positive stopping times.

Remark 27*.*

If we take $\tau=1$, we observe by boundedness of $h$ (Assumption 3.1) that $\gamma(t)<C$.

To study the process $\gamma$, we introduce an auxiliary optimal stopping problem. At each time step, the player decides whether to continue or to stop play of the machine. If the player decides to continue to play, he will be offered a fixed reward $\lambda$ (known at the initial time $s$) in addition to the cost $h(t)$.

Definition A.1**.**

The target function $V_{s}:\mathcal{T}(s)\times L^{\infty}(\mathcal{F}_{s})\to L^{\infty}(\mathcal{F}_{s})$ for a stopping time $\tau\in\mathcal{T}(s)$ with a reward $\lambda$ is defined by

[TABLE]

We know that $\gamma(s)$ is defined to be the minimum reward $\lambda$ such that, with a choice of $\tau$ minimizing $V_{s}(\tau,\lambda)$, the expected loss is at most zero. By minimality of $\gamma(s)$ and monotonicity of $\mathcal{E}$, the reward $\gamma(s)$ will yield zero loss under optimal stopping and, therefore, cannot yield a positive expected reward under suboptimal stopping. In particular, the following holds.

Theorem A.2**.**

The function $V_{s}$ defined above satisfies.

[TABLE]

Proof.

This can be done by showing that $V_{s}$ satisfies the regularity assumptions of Lemma C.6.

By considering $\lambda=C+1$, where $C$ is an upper bound on $h$, we see that $V_{s}(\tau,C+1)<0$. As $h(t)\geq 0$, it also follows that $V_{s}(\tau,0)\geq 0$. Hence, condition (i) is satisfied.

For condition (ii), suppose that $\lambda^{\prime}>\lambda$. Then

[TABLE]

By monotonicity and translation equivariance, we have

[TABLE]

So, $V_{s}$ is Lipschitz in $\lambda$.

Condition (iii) follows from $(\mathcal{F}_{t})_{t\geq 0}$-regularity of $\mathcal{E}$ (Remark 8). The result follows from Lemma C.6. ∎

Corollary A.3**.**

For every $\tau\in\mathcal{T}(s)$, we have

[TABLE]

Remark 28*.*

Theorem A.2 shows that, under optimal stopping, with the reward $\gamma(s)$, the expected total loss is zero. In particular, we may view $\gamma(s)$ as an ‘average cost under optimal play’ of the bandit.

A.2 Step A.2: Optimal Stopping time

By considering a Snell envelope argument, as in Riedel [68] with slight modification, we can establish that a stopping time $\tau^{*}$ achieving the minimum value $V_{s}(\tau^{*},\lambda)=\operatorname*{ess\,inf}_{\tau\in\mathcal{T}(s)}V_{s}(\tau,\lambda)$ exists (Theorem C.7). In this subsection, we will show that $\tau^{*}$ can be expressed as a hitting time of the Gittins index process $(\gamma(s))$.

Definition A.4**.**

Let $\lambda$ be a non-negative $\mathcal{F}_{s}$-measurable random variable. Define a stopping time $\sigma(s,\lambda)$ by

[TABLE]

As mentioned in Remark 28, we may view $\gamma$ as a time-average cost under optimal stopping. The stopping time $\sigma(s,\lambda)$ can be interpreted as the first time when this average cost exceeds a fixed $\lambda$. Once $\gamma$ exceeds $\lambda$, the offered compensation $\lambda$ is insufficient to make the bandit attractive so, to minimize the total ‘expected’ cost, we will stop.

In what follows, we formalize this intuition. We will show that $\sigma(s,\lambda)$ is an optimal stopping time when the reward $\lambda$ is offered. In particular, we will show that $\sigma(s,\gamma(s))$ attains the optimal value with the reward $\lambda=\gamma(s)$. Moreover, the value for this optimal stopping problem is zero (by Theorem A.2).

The optimality of $\sigma(s,\lambda)$ can be proved by showing that for any stopping time $\tau\in\mathcal{T}(s)$, if $\tau>\sigma(s,\lambda)$ on some event, our value can be improved by stopping at $\sigma(s,\lambda)$ (Lemma A.5). On the other hand, if $\tau<\sigma(s,\lambda)$ on some event, the value can be improved by continuing to play (Lemma A.6). The easy proofs of these Lemmata are in the appendix C.

Lemma A.5**.**

For every $\lambda\in L^{\infty}(\mathcal{F}_{s})$ taking values in $[0,C)$ and $\tau\in\mathcal{T}(s)$,

[TABLE]

Proof.

We will prove this result by applying Corollary A.3 together with time-consistency and monotonicity of our nonlinear expectation.

Define $\nu=\tau\wedge\sigma(s,\lambda)$. By Corollary A.3 and regularity (Remark 8),

[TABLE]

By translation equivariance,

[TABLE]

By monotonicity and time-consistency,

[TABLE]

In particular, $V_{s}(\tau\wedge\sigma(s,\lambda),\lambda)=V_{s}(\nu,\lambda)\leq V_{s}(\tau,\lambda)$. ∎

Lemma A.6**.**

Let $\tau\in\mathcal{T}(s)$ and let $\lambda\in L^{\infty}(\mathcal{F}_{s})$ taking values in $[0,C)$. Then there exists a stopping time $\tau_{1}\in\mathcal{T}(s)$ with $\tau_{1}\geq\tau$ such that

[TABLE]

and on the event $A:=\{\gamma(s+\tau)\leq\lambda\}$, we have $\tau_{1}>\tau$.

Proof.

For $A=\{\gamma(s+\tau)\leq\lambda\}$, define $\gamma^{\tau}:=\gamma(s+\tau)\mathbb{I}_{A^{c}}+\lambda\mathbb{I}_{A}\geq\gamma(s+\tau)$. By Theorem A.2 and monotonicity of our nonlinear expectation,

[TABLE]

Thus, by Theorem C.7, there exists $\tilde{\tau}^{*}\in\mathcal{T}(s+\tau)$ such that,

[TABLE]

Define a stopping time $\tau_{1}:=\tau+\tilde{\tau}^{*}\;\mathbb{I}_{A}$. As $\gamma^{\tau}\mathbb{I}_{A}=\lambda\mathbb{I}_{A}$, then

[TABLE]

By translation equivariance and regularity (Remark 8), it follows that

[TABLE]

Finally, by applying monotonicity and time-consistency as in the previous lemma, the result follows. ∎

Corollary A.7**.**

Let $\tau\in\mathcal{T}(s)$. Then there exists an increasing sequence $(\tau_{n})_{n\geq 1}$ in $\mathcal{T}(s)$ with $\tau_{n+1}\geq\tau_{n}\geq\tau$ for all $n\geq 1$ such that

[TABLE]

and on the event $\bigcap_{k=1}^{n-1}\left\{\gamma(s+\tau_{k})\leq\lambda\right\}$ we have $\tau_{n}>\tau_{n-1}>...>\tau_{1}>\tau$. In particular, on this event, $\tau_{n}\geq n$.

By combining these observations with the Lebesgue property of $\mathcal{E}$, we have the following theorem.

Theorem A.8**.**

For every $\lambda\in L^{\infty}(\mathcal{F}_{s})$ taking values in $[0,C)$ and $\tau\in\mathcal{T}(s)$, we have

[TABLE]

Therefore,

[TABLE]

In particular, $\sigma(s,\gamma(s))$ yields equality in Corollary A.3.

Proof.

By Lemma A.5 and Corollary A.7,

[TABLE]

Observe that by Corollary A.7,

[TABLE]

Hence, it follows that $\tau_{n}\wedge\sigma(s,\lambda)\to\sigma(s,\lambda)$ as $n\to\infty$ and thus

[TABLE]

As $h$ is bounded, it follows from the Lebesgue property of our nonlinear expectation that

[TABLE]

In particular, $V_{s}(\tau,\lambda)\geq V_{s}(\sigma(s,\lambda),\lambda).$ ∎

Remark 29*.*

Bank and El Karoui [9] consider a similar result to this theorem, but under a classical expectation with the summation $\sum_{t=s+1}^{s+\tau}\beta^{t}\left(h(t)-\gamma(s)\right)$ replaced by a more general function in continuous time. (See also [10] and [11] for further discussion).

A.3 Step A.3: Fair Game and Prevailing process

Previously, we considered an optimal stopping problem when the Gittins index is offered as compensation for continued play. In this subsection, we consider a ‘fair game’ when we offer a compensation which is (just) sufficient to encourage us to continue playing the bandit. In particular, the compensation increases at each optimal stopping time in order to encourage the agent to continue.

We will first define a sequence of optimal stopping times that we have to consider in order to analyze our (minimal) compensation process.

Definition A.9**.**

We define $\hat{S}_{n}$ to be the stopping time where the Gittins index process $\left(\gamma(s)\right)_{s\geq 0}$ exceeds its running maximum for the $n$th time. We write $\sigma_{n}$ for the duration between $\hat{S}_{n}$ and $\hat{S}_{n+1}$, that is, $\sigma_{n}$ is a random time identifying how long after time $\hat{S}_{n}$ the process $\left(\gamma(s)\right)_{s\geq 0}$ hits a new maximum.

More precisely, we define $\hat{S}_{n}$ and $\sigma_{n}$ inductively:

1. (i)

Let $\hat{S}_{0}:=0$. 2. (ii)

Given $\hat{S}_{n}$, define

[TABLE]

and $\hat{S}_{n+1}:=\hat{S}_{n}+\sigma_{n}$.

Equivalently, we can define $\sigma_{n}:=\sigma(\hat{S}_{n},\gamma(\hat{S}_{n}))$ as in Definition A.4.

Definition A.10**.**

We define the prevailing reward process $\Gamma$ by the running maximum of $\gamma$, that is,

[TABLE]

We can then show that the process $\Gamma$ serves as an indifference reward (process) for our agent, when evaluated from the perspective of one of the stopping times $\hat{S}_{n}$.

Proposition A.11**.**

For all $n\in\mathbb{N}$,

[TABLE]

In particular,

[TABLE]

Proof.

By Theorem A.8, we have, for all $k\in\mathbb{N}$,

[TABLE]

Fix $n,N\in\mathbb{N}$ with $N\geq n$, by time-consistency, translation equivariance,

[TABLE]

By our definition of $\hat{S}_{n}$, we have $\hat{S}_{N}\geq N$. Hence, $\hat{S}_{N}\to\infty$ as $N\to\infty$. Therefore, by applying Lebesgue property, the result follows. ∎

Intuitively, as $\Gamma(t)\geq\gamma(t-1)$, the process $\Gamma$ should be sufficient to compensate for continuing to play. This means that the total ‘expected’ loss, evaluated from any point in time, must be non-positive if a reward $\Gamma(t)$ is offered. This is stated formally in the following lemma and theorem.

Lemma A.12**.**

Let $\tau\in\mathcal{T}(s)$ with $1\leq\tau\leq\sigma:=\sigma(s,\lambda)$. Then

[TABLE]

Proof.

Write H_{\tau}:=\mathcal{E}\left(\sum_{t=s+\tau+1}^{s+\sigma}\beta^{t}\Big{(}h(t)-\lambda\Big{)}\Big{|}\mathcal{F}_{s+\tau}\right) and $A:=\left\{H_{\tau}>0\right\}$.

Define $\tilde{\sigma}:=\tau\;\mathbb{I}_{A}+\sigma\;\mathbb{I}_{A^{c}}$.

[TABLE]

Moreover, the above inequality is strict on $A$. Hence, if $A$ is not a $\mathbb{P}$-null set, it then follows from strict monotonicity that

[TABLE]

This contradicts the minimality of $\sigma(s,\lambda)$ established in Theorem A.8. ∎

Theorem A.13**.**

For all $N\in\mathbb{N}$,

[TABLE]

Proof.

Define $\tau_{n}:=(\hat{S}_{n+1}\wedge N)\vee\hat{S}_{n}$. Since $\hat{S}_{n}$ is a stopping time for all $n\in\mathbb{N}$, so is $\tau_{n}$. Hence, by Proposition A.11 and Lemma A.12,

[TABLE]

Therefore, as $\{\hat{S}_{n}\leq N<\hat{S}_{n+1}\}$ is $\mathcal{F}_{N}$-measurable, by Lebesgue property and regularity (Remark 8),

[TABLE]

∎

Remark 30*.*

The above theorem says that, with compensation $\Gamma(t)$, at any point in time we expect to obtain a net reward from continuing to play, i.e. we have a non-positive expected total loss.

A.4 Step A.4: Reward Delay and Robust Representation Theorem

In Step A.3, we have shown that our reward $\Gamma$ is defined to be (just) sufficient to encourage the player to continue playing (Theorem A.13) until the horizon (i.e. the total expected loss is zero, as in Proposition A.11). We now show that taking a break from play cannot improve a player’s expected discounted costs. We now formulate this observation by establishing the existence of a probability measure in our representing set $\mathcal{Q}$ such that the expected discounted costs, accounting for the break in play, have a lower bound close to zero. This result will be useful when considering multiple bandits.

Theorem A.14**.**

By Assumption 3.6, recall that $\mathcal{E}$ admits a robust representation of the form

[TABLE]

For every fixed $\epsilon>0$, there exists a probability measure $\mathbb{Q}\in\mathcal{Q}$ such that for every predictable decreasing process $(\alpha(t))_{t\geq 0}$ taking values in $[0,1]$, we have

[TABLE]

Proof.

By Proposition A.11 and the robust representation theorem, for a fixed $\epsilon>0$, we can find a probability measure $\mathbb{Q}\in\mathcal{Q}$ such that

[TABLE]

For each predictable decreasing process $(\alpha(t))_{t\geq 0}$ taking values in $[0,1]$, we define

[TABLE]

We claim that

[TABLE]

Indeed, it is clear that the result holds when $N=0$.

For the sake of induction, assume that the result holds for a given $N$. We then have

[TABLE]

By the robust representation theorem (Theorem 3.5),

[TABLE]

By Theorem A.13, we know that

[TABLE]

Since $\alpha$ is decreasing,

[TABLE]

As $\alpha$ is predictable, by rearranging the above inequality, we obtain

[TABLE]

Hence, by the tower property,

[TABLE]

By substituting this into (A.2), we prove (A.1) with $N$ replaced by $N+1$ and done the induction step.

By using bounded convergence theorem, we can take $N\to\infty$ and obtain the required result. ∎

Appendix B Part B: Analysis of multiple bandits

We are now ready to consider the problem of choosing between multiple bandits.

In Definition 4.6, we introduce our class of admissible control which can be considered in our dynamic allocation problems. This class of control introduces a few natural ways of parameterizing time. We therefore will use the following terminology to describe the evolution of time in different ways. This terminology will be useful in our discussion on the proof.

‘Play’ refers to the total number of (real) times that we play the system of bandits. (This corresponds to the time parameter of the simple form $\rho$ which is briefly discussed earlier on in Remark 2 and later in Definition 4.8.) 2. 2.

‘Trial’ refers to the number of times that we play a specific bandit. (This corresponds to the sum $\sum_{k=0}^{K}\tau^{(m)}_{k}$.) 3. 3.

‘Decision’ refers to the number of times that we make a decision between bandits. (This corresponds to the time parameter for the choice process $\left(p_{n}\right)_{n\geq 0}$.) 4. 4.

‘Run’ refers to the number of times that we have made the decision to select a specific bandit. (This corresponds to the time parameter of the allocation sequence $(\tau^{(m)}_{k})_{k\geq 0}$ for each fixed $m\in\mathcal{M}$.)

Remark 31*.*

The terms ‘play’ and ‘trial’ can be referred to without directly identifying the time allocation sequence. On the other hand, the terms ‘decision’ and ‘run’ need to be interpreted under a given time allocation sequence $\tau$ (Definition 4.4).

Remark 32*.*

The $m$th component of the recording sequence $\eta_{n}$ (Definition 4.5) represents the number of runs in the $m$th bandit before the $n$th decision. We can see that the random variable $\eta_{n}$ takes values in \Big{\{}r\in\mathcal{S}:\sum_{m=1}^{M}r^{(m)}\leq n\Big{\}}.

In order to prove C-optimality, we then consider the target function as briefly stated in (4.9).

Definition B.1**.**

For each $m\in\mathcal{M}$, let \big{(}h^{(m)}(t)\big{)}_{t\geq 1} be the uniformly bounded non-negative cost process at the $t$th trial of the $m$th bandit with prevailing reward process \big{(}\Gamma^{(m)}(t)\big{)}_{t\geq 1} (Definition A.10). For an allocation strategy $(\tau,p)$, (Definition 4.6), we define the Gittins’ target function by

[TABLE]

where $\rho$ is the simple form of $({\tau},p)$ with corresponding counting processes $t^{\rho}_{n}:=\sum_{k=0}^{n-1}\mathbb{I}(\rho_{k}=\rho_{n-1})$, and $\mathfrak{E}$ is a partially consistent orthant nonlinear expectation, as in Definition 3.7.

Remark 33*.*

We can also write $V({\tau},p)$ in terms of ${\tau}$ and $p$ directly without identifying the simple form ${\rho}$. This is done in the proof of Theorem B.4 in Step B.2. This definition, however, makes it clear that $V$ depends on $(\tau,p)$ only through its simple form.

B.1 Step B.1: Fubini theorem and Suboptimality

In this subsection, we will show that considering generic stopping times and choice of bandits yields a non-negative expected loss. This can be shown using the robust representation result.

First, we recall the following corollary of Fubini’s theorem.

Corollary B.2**.**

Let $(G,\mathcal{G},\mathbb{P})$ and $(H,\mathcal{H},\mathbb{Q})$ be probability spaces. Let $\mathcal{G}^{\prime}$ and $\mathcal{H}^{\prime}$ be sub $\sigma$-algebras of $\mathcal{G}$ and $\mathcal{H}$ respectively, with $\mathcal{H}^{\prime}:=\{\emptyset,H\}$. Then, for any integrable random variable $X$ on $(G\times H,\mathcal{G}\otimes\mathcal{H},\mathbb{P}\otimes\mathbb{Q})$, we have

[TABLE]

Theorem B.3**.**

For any allocation strategy $(\tau,p)$, we have

[TABLE]

Proof.

Let $\rho$ be the simple form of $(\tau,p)$. Write $R^{(m)}_{t}$ for the total number of trials on other bandits before making the $t$th trial on the $m$th bandit, i.e.

[TABLE]

Since $R^{(m)}_{t}$ does not depend on future realizations of the $m$th bandit, $R^{(m)}_{t}$ is \mathcal{F}^{(m)}_{t-1}\otimes\big{(}\bigotimes_{k=1,k\neq m}^{M}\mathcal{F}^{(k)}_{\infty}\big{)}-measurable (taking the product in an appropriate order). Moreover, as $N^{(m)}_{t}$ is increasing in $t$, it follows that $R^{(m)}_{t}$ is increasing in $t$.

Now, fix $\epsilon>0$. By Theorem A.14, for each $m\in\mathcal{M}$, we can find a probability measure $\mathbb{Q}^{(m)}\in\mathcal{Q}^{(m)}$ such that, for every adapted decreasing process $(\alpha^{(m)}(t))_{t\geq 0}$ taking values in $[0,1]$, we have

[TABLE]

Define

[TABLE]

By Fubini’s theorem, as $R^{(m)}_{t}$ is \mathcal{F}^{(m)}_{t-1}\otimes\big{(}\bigotimes_{k=1,k\neq m}^{M}\mathcal{F}^{(k)}_{\infty}\big{)}-measurable and $\beta\in(0,1]$, the process \big{(}\tilde{\alpha}^{(m)}(t)\big{)} is an \big{(}\mathcal{F}^{(m)}_{t}\big{)}-predictable process taking values in $[0,1]$.

Moreover, $R^{(m)}_{t}$ is also \mathcal{F}^{(m)}_{\infty}\otimes\big{(}\bigotimes_{k=1,k\neq m}^{M}\mathcal{F}^{(k)}_{\infty}\big{)}-measurable, so by Corollary B.2 we can write

[TABLE]

As $t\mapsto R^{(m)}_{t}$ is increasing, it then follows that $\tilde{\alpha}^{(m)}(t)$ is decreasing in $t$. Hence, by Theorem A.14 we obtain (B.1) with $\alpha^{(m)}$ replaced with $\tilde{\alpha}^{(m)}$.

By the definition of $\mathfrak{E}$ and Fubini’s theorem, it follows that

[TABLE]

As $\epsilon$ is arbitrary, the result follows. ∎

B.2 Step B.2: Optimality

In this subsection, we will show that the strategy determined by a particular time allocation sequence yields a zero expected cost in the Gittins’ target function.

Theorem B.4**.**

For each $m\in\mathcal{M}$, let $(\sigma^{(m)}_{k})_{k\geq 0}$ be the sequence of running maximum random times associated to the $m$th bandit, as defined in Definition A.9, i.e. we define $\sigma^{(m)}_{k}$ and $\hat{S}^{(m)}_{k}$ recursively by $S^{(m)}_{k}:=\sum_{l=0}^{k-1}\sigma^{(m)}_{l}$ and

[TABLE]

Then for any allocation strategy of the form $(\sigma,p)$, we have

[TABLE]

Proof.

Recall the recording sequence $\eta_{n}$ associated with $(\sigma,p)$. We define the following notation, given an allocation sequence $\sigma$.

1. $\bullet$

$\tilde{\Theta}_{n}$ denotes the total number of plays of the system before making the $n$th decision. i.e.

[TABLE] 2. $\bullet$

$\tilde{\sigma}_{n}$ denotes the duration we decide to play following the $n$th decision time, i.e.

[TABLE]

N.B. $p_{n}=m$ means that we decide to play the $m$th bandit at the $n$th decision. The event $\eta^{(m)}_{n}=k$ means that we have had $k$ runs of the $m$th bandit before the $n$th decision. Thus, we choose to make $\sigma^{(m)}_{k}$ more trials on this bandit before making another decision. 3. $\bullet$

$\tilde{\Psi}^{(m)}_{n}$ denotes the total number of trials on the $m$th bandit before making the $n$th decision, i.e.

[TABLE]

Using this notation, we can define a variation on the Gittins’ target fuction, with the restriction that we consider only the first $N$ plays of the system, that is,

[TABLE]

with the convention $h^{(0)}(t)=\Gamma^{(0)}(t)=0$ for all $t$.

By considering the simple form $\rho$ of the strategy $(\sigma,p)$ and applying Lebesgue property of $\mathfrak{E}$, we can show that $V$ agrees with Definition B.1 as $N\to\infty$, that is,

[TABLE]

Hence, it suffices to show that $V(N,\sigma,p)\leq 0$ for all $N\in\mathbb{N}$. This will be proved by induction.

It is clear that $V(0,\sigma,p)=0$. Fix $N\in\mathbb{N}$ and assume that $V(N,\sigma,p)\leq 0$. To show that $V(N+1,\sigma,p)\leq 0$, by subadditivity, it suffices to show that

[TABLE]

Define the following random variables as in (4.7):

[TABLE]

for

[TABLE]

Note that $\Psi_{r}^{(m)}=\tilde{\Psi}_{N}^{(m)}$ and $\Theta_{r}=\tilde{\Theta}_{N}$ on the event $\{\eta_{N}=r\}$. From the definition of the recording sequence (Definition 4.5), it follows that $\Theta_{r}$ and the event $A_{N}^{(r,m)}:=\{\eta_{N}=r,p_{N}=m\}$ are both $\mathcal{F}(\Psi_{r})$-measurable.

On an event $A_{N}^{(r,m)}$, $\tilde{\Psi}_{N}^{(m)}=\Psi_{r}^{(m)}$ and $\Psi_{r}^{(m)}$ is a stopping time with respect to the filtration $(\mathcal{F}^{(m)}_{t})_{t\geq 0}$. By considering the optimality obtained in Theorem A.8, we can show that

[TABLE]

We see that $V(N+1,\sigma,p)\leq 0$, and the desired result follows by induction. ∎

B.3 Step B.3: C-optimality

In the previous subsections, we introduced an allocation problem when the prevailing process is offered as compensation (Definition B.1). We also proved that the optimal value can be achieved by choosing a proper family of allocation time sequences (i.e. $\sigma$ as in Theorem B.4).

The prevailing reward process $\Gamma$ for each bandit is non-decreasing, and the optimal allocation sequences $\sigma$ require us to make a new decision whenever the process $\Gamma$ increases. By exploiting this fact, together with the discount effect, we will see that it is preferable to play the bandit with the lowest value of $\Gamma$ first. In particular, we can establish the Robust Gittins index theorem, which we repeat for convenience of the reader.

Theorem B.5** (Theorem 4.3: Robust Gittins theorem).**

Suppose that for each $m\in\mathcal{M}$, $(\mathcal{F}^{(m)}_{t})_{t\geq 0}$ is generated by some underlying process $(\xi^{(m)}_{t})_{t\geq 1}$. Let $\psi^{(m)}_{n}$ be the total number of trials of the $m$th bandit before the $n$th play of the system. i.e. $\psi^{(m)}_{n}:=\sum_{k=0}^{n-1}\mathbb{I}(\rho^{*}_{k}=m)$ (given an allocation strategy $\rho^{*}$ up to time $n-1$).

Then the allocation strategy $\rho^{*}$ given (recursively) by

[TABLE]

is C-optimal (Definition 3.12) under $\mathfrak{E}$ for the cost

[TABLE]

Proof.

Recall the definition of $\Psi^{(m)}_{r}$ in (4.7). We can see that $\Psi_{r}$ determines the orthant filtration when $\tilde{\eta}_{n}=r$ where $(\tilde{\eta}_{n})$ is a recording sequence constructed from the time allocation sequence $\sigma$, i.e. when the $m$th bandit was run for $r^{(m)}$ times under the (optimal) allocation sequence $\sigma$. In particular, $\Psi^{(m)}_{r}$ corresponds to the number of trials on the $m$th bandit.

To explicitly define our choice sequence, we set

[TABLE]

As $\Psi^{(m)}_{r}$ is an $(\mathcal{F}^{(m)}_{t})$-stopping time, $\gamma^{(m)}(\Psi^{(m)}_{r})$ is well-defined and is $\mathcal{F}(\Psi^{(m)}_{r})$-measurable. It also follows that

[TABLE]

Hence, $p^{*}$ is a choice sequence for the allocation sequence $\sigma$ (Definition 4.5). Therefore, $(\sigma,p^{*})$ is an admissible allocation strategy. Moreover, observe that $\rho^{*}$ given in the statement of this theorem is the simple form of the allocation strategy $(\sigma,p^{*})$.

By Theorem B.4 and Theorem B.3,

[TABLE]

Theorem B.3 also implies that for any allocation strategy $(\tau,p)$ (and thus for any simple form $\rho$),

[TABLE]

Next, we will show that $n\mapsto\beta^{n}\Gamma^{(\rho_{n-1})}(t^{\rho}_{n})$ is predictable with respect to our observed filtration. We recall that

[TABLE]

Now, observe that $t^{\rho}_{n}=\sum_{k=0}^{n-1}\mathbb{I}(\rho_{k}=\rho_{n-1})=1+\eta^{(\rho_{n-1})}_{n-1}$ where $(\eta_{n})$ is a recording sequence corresponding to a strategy $(\mathfrak{1},\rho)$ and hence,

[TABLE]

where {\mathcal{S}}_{N}:=\Big{\{}r\in\mathcal{S}\;:\;\sum_{m=1}^{M}r^{(m)}=N\Big{\}}.

Since $\Gamma^{(m)}(1+r^{(m)})$ is $\mathcal{F}^{(m)}_{r^{(m)}}$-measurable, by the Doob–Dynkin lemma, there exists a measurable function $f^{(m)}_{r}:\mathbb{R}^{r^{(m)}}\to\mathbb{R}$ such that

[TABLE]

Note that, while the process \big{(}\xi^{(m)}_{t}\big{)} is defined on $(\Omega^{(m)},\mathcal{F}^{(m)})$, we can extend it to $(\bar{\Omega},\bar{\mathcal{F}})$ by considering an appropriate embedding.

By substituting into (B.3) in a similar way to (C.6), we can write $\Gamma^{(\rho_{n-1})}(t^{\rho}_{n})$ as a (measurable) function of $\left(\left(\eta_{k}\right)_{0\leq k\leq n-1},\left(\rho_{k}\right)_{0\leq k\leq n-1},\left(\xi^{\rho}_{k}\right)_{0\leq k\leq n-1}\right)$. By Definition 4.5 and Remark 23, $\left(\eta_{k}\right)$ and $\left(\rho_{k}\right)$ are adapted to the observed filtration $\left(\mathcal{H}^{\rho}_{n}\right)_{n\geq 0}$. It follows that $\Gamma^{(\rho_{n-1})}(t^{\rho}_{n})$ is $\mathcal{H}^{\rho}_{n-1}$-measurable.

Therefore, for each $\rho$, $C_{0}^{\rho}(n):=\Gamma^{(\rho_{n-1})}(t^{\rho}_{n})$ defines a subcompensator at time $N=0$, and for $\rho=\rho^{*}$, the cost is fully compensated with $V_{0}=0$.

To construct a compensator for a subsequent time $N$, we consider ‘restarting’ our system at an orthant time $r=(r^{(1)},...,r^{(M)})\in{\mathcal{S}}_{N}\subseteq\mathcal{S}$ (as in Theorem C.12). As $\mathcal{F}(r)$ describes the information from all bandits, this needs to be done carefully. Each of our single-bandit filtrations $(\mathcal{F}^{(m)}_{t})_{t\geq 0}$ is generated by a discrete-time real-valued process, and $\mathcal{F}(r)=\bigotimes_{m}\mathcal{F}^{(m)}_{r^{(m)}}$, so the Doob–Dynkin lemma states that any $\mathcal{F}(r)$-measurable random variable can be written as a Borel function of the first $r$ observations. For concreteness, we denote these observations $\omega_{r}$.

We proceed by freezing the value of $\omega_{r}$ and $\omega_{u}$ with $u\leq r$. Let $(\rho^{*,\omega_{r}}_{n})_{n\geq N}$ denote the minimum-Gittins-index strategy given by $\rho^{*}$ defined in the theorem when we restart our analysis at $r$ from a given $\omega_{r}$. We do not change the Gittins indices $\gamma$ when we fix $\omega_{r}$, so the corresponding $\Gamma$ processes satisfy

[TABLE]

and are measurable with respect to $\omega_{r}$. As discussed in Remark 19, the optimal strategy $(\rho^{*,\omega_{r}}_{n})_{n\geq N}$ coincides with the strategy $\rho^{*}$ (and is therefore also measurable with respect to $\omega_{r}$).

By repeating our earlier analysis, we see that, for each $\omega_{r}$, $\Gamma_{\omega_{r}}^{(\rho^{*,\omega_{r}}_{n-1})}\leq\Gamma_{\omega_{u}}^{(\rho^{*,\omega_{r}}_{n-1})}$. We can now unfreeze $\omega_{r}$ and $\omega_{u}$ and, summing over all possible scenarios, show that

[TABLE]

is decreasing in $N$. By applying the same argument as earlier, we also have $n\mapsto C^{\rho}_{N}(n)$ is $\mathcal{H}^{\rho}_{n}$-predictable. Therefore, $(C^{\rho}_{N}(n))$ defines a subcompensator for strategy $\rho$ and fully compensates for $\rho=\rho^{*}$. We set $V_{N}=0$ and observe (3.2) is satisfied.

Finally, as $t\mapsto\Gamma_{\omega_{r}}^{(m)}(t)$ is increasing for all $m$ and all $\omega_{r}$, and $\rho^{*}$ is a strategy where the lowest $\Gamma$ is chosen first, it follows that for all $1\leq N\leq\infty$, if $\rho\sim_{N}\rho^{*}$, then

[TABLE]

In particular, (3.3) is satisfied and therefore $\rho^{*}$ is C-optimal. ∎

Appendix C Proof of Other Relevant Results

Definition C.1** (Föllmer and Schied [36]).**

Let $(\Omega,\mathcal{G},\mathbb{P})$ be a probability space and let $\mathcal{Y}$ be a family of $\mathcal{G}$-measurable random variable. We say $Z$ is a $\mathcal{G}$-essential infimum of $\mathcal{Y}$ denoted by $Z=\mathcal{G}\text{-}\operatorname*{ess\,inf}Y$ if

1. (i)

$Z$ is $\mathcal{G}$-measurable. 2. (ii)

$Z\leq Y$ $\;\mathbb{P}-a.s.$ for all $Y\in\mathcal{Y}$. 3. (iii)

For $Z^{\prime}$ such that $Z^{\prime}\leq Y$ $\;\mathbb{P}-a.s.$ for all $Y\in\mathcal{Y}$, we must have $Z^{\prime}\leq Z$ $\mathbb{P}-a.s.$.

We also define a similar notion for $\mathcal{G}$-essential supremum. We may omit $\mathcal{G}$ in front of $\operatorname*{ess\,inf}$ if the measurability of the family is obvious.

Theorem C.2** (Existence of Essential infimum).**

The $\mathcal{G}$-essential infimum exists.

Suppose in addition that $\mathcal{Y}$ is directed downwards, that is for $Y,Y^{\prime}\in\mathcal{Y}$, there exists $\tilde{Y}\in\mathcal{Y}$ such that ${\tilde{Y}}\leq\min(Y,Y^{\prime})$. Then there exists a decreasing sequence $\left(Y_{n}\right)_{n\in\mathbb{N}}\subseteq\mathcal{Y}$ such that $Y_{n}\searrow\operatorname*{ess\,inf}Y$ $\;\mathbb{P}$-a.s.

The similar result also holds for $\mathcal{G}$-essential supremum.

Proof.

See Theorem 1.3.40. in [29] or Theorem A.37 in [36]. ∎

Theorem C.3** (Proof of Theorem 4.2).**

Let $\gamma(s)$ be the robust Gittins index (Definition 4.1) (with superscript $(m)$ omitted). Then

[TABLE]

where $\mathcal{Q}$ is a family of probability measure induced in Theorem 3.5.

Proof.

By Corollary A.3 and Theorem 3.5 (together with positive homogeneity), we have that, for any $\tau\in\mathcal{T}(s)$,

[TABLE]

By Lemma 11.19 (together with the construction of Theorem 11.22) in Föllmer and Schied [36], the family \big{\{}\mathbb{E}^{\mathbb{Q}}\big{(}\sum_{t=1}^{\tau}\beta^{t}\big{(}h(s+t)-\gamma(s)\big{)}\big{|}\mathcal{F}_{s}\big{)}:\mathbb{Q}\in\mathcal{Q}\big{\}} must be directed upwards. Hence, by Theorem C.2, we can find a family $(\mathbb{Q}_{n})\subseteq\mathcal{Q}$ such that

[TABLE]

Let $\tilde{\Omega}$ be an event with probability one such that the followings hold.

(C.1) and (C.2) holds. 2. 2.

For all $n\in\mathbb{N}$,

[TABLE]

\mathbb{E}^{\mathbb{Q}_{n}}\big{(}\sum_{t=1}^{\tau}\beta^{t}\big{|}\mathcal{F}_{s}\big{)}\geq\beta, and

[TABLE]

Fix $\omega\in\tilde{\Omega}$ and $\epsilon>0$. By (C.1) and (C.2), there exists $n\in\mathbb{N}$ such that

[TABLE]

By (2), we can rearrange the inequality above and obtain

[TABLE]

As $\epsilon$ is arbitrary, it follows that on $\tilde{\Omega}$,

[TABLE]

hence,

[TABLE]

By Theorem A.8 and Theorem 3.5, we can find $\sigma:=\sigma(s,\gamma(s))\in\mathcal{T}(s)$ such that,

[TABLE]

hence, for all $\mathbb{Q}\in\mathcal{Q}$,

[TABLE]

Therefore,

[TABLE]

and we conclude

[TABLE]

This completes the proof. ∎

Proposition C.4**.**

All properties described in Proposition 3.8 hold.

Proof.

$(i)$ is straightforward to prove using the definition directly, as in the case of $\mathcal{E}$.

By Theorem 3.5 and the tower property,

[TABLE]

hence $(ii)$ follows.

For $\hat{Y}(\omega^{(1)},...,\omega^{(M)})=X^{(1)}(\omega_{1})\times\cdots\times X^{(M)}(\omega_{M})$, by Fubini’s theorem,

[TABLE]

Then $(iii)$ and $(iv)$ follow by considering different choices of $X^{(m)}$. ∎

Lemma C.5**.**

Let $V_{s}:\mathcal{T}(s)\to L^{\infty}(\mathcal{F}_{s})$ be a function such that for every $\tau,\sigma\in\mathcal{T}(s)$ and $A\in\mathcal{F}_{s}$, we have

[TABLE]

Then there exists a sequence $\tau_{n}\in\mathcal{T}(s)$ such that $V_{s}(\tau_{n})\searrow\operatorname*{ess\,inf}_{\tau\in\mathcal{T}(s)}V_{s}(\tau)$ $\mathbb{P}$-a.s..

Proof.

For $\tau,\sigma\in\mathcal{T}(s)$, we define $A:=\left\{V_{s}(\tau)>V_{s}(\sigma)\right\}\in\mathcal{F}_{s}$. Then

[TABLE]

By assumption, $V_{s}(\tilde{\tau})\geq V_{s}(\tau)\wedge V_{s}(\sigma)$; the result follows from Theorem C.2. ∎

Lemma C.6**.**

Let $f:\mathcal{T}(s)\times L^{\infty}(\mathcal{F}_{s})\to L^{\infty}(\mathcal{F}_{s})$ satisfy:

1. (i)

For all $\tau\in\mathcal{T}(s)$, $f(\tau,0)\geq 0$ $\mathbb{P}$-a.s. and $f(\tau,X)<0$ for some $X\in L^{\infty}(\mathcal{F}_{s}).$ 2. (ii)

There exists $L\in[0,\infty)$ such that, for every $X,Y\in L^{\infty}(\mathcal{F}_{s})$ with $X\geq Y$ $\mathbb{P}$-a.s and $\tau\in\mathcal{T}(s)$, we have

[TABLE] 3. (iii)

For all $A\in\mathcal{F}_{s}$, all $\tau,\sigma\in\mathcal{T}(s)$ and all $X,Y\in L^{\infty}(\mathcal{F}_{s})$, we have

[TABLE]

Define

[TABLE]

Then $X^{*}\in L^{\infty}(\mathcal{F}_{s})$ and

[TABLE]

Note: All essential infima in this lemma are taken among the $\mathcal{F}_{s}$-measurable functions.

Proof.

Denote

[TABLE]

By (i), $\mathcal{X}\neq\phi$. For a fixed $X\in\mathcal{X}$, by Lemma C.5 there exists a sequence $\tau_{k}\in\mathcal{T}(s)$, such that $f(\tau_{k},X)\searrow\operatorname*{ess\,inf}_{\tau\in\mathcal{T}(s)}f(\tau,X)$ $\mathbb{P}\text{-a.s.}$ Similarly, we can find a sequence $(\tau^{\prime}_{k})$ for $X^{\prime}\in\mathcal{X}$.

Define a sequence $\sigma_{k}:=\tau_{k}\mathbb{I}_{A}+\tau^{\prime}_{k}\mathbb{I}_{A^{c}}$ where $A=\{X\leq X^{\prime}\}$. Then

[TABLE]

Hence, $\mathcal{X}$ is downward directed. Therefore, by Theorem C.2, there exists a sequence $\left(X_{n}\right)_{n\geq 0}\subseteq\mathcal{X}$ such that $X_{n}\searrow X^{*}$ $\mathbb{P}$-a.s. This implies that $X^{*}$ is almost surely bounded from above. By monotonicity, as $f(\tau,0)\geq 0$, it follows from strict monotonicity that $f(\tau,-1)>0$. Therefore, $-1$ is an essential lower bound of $\mathcal{X}$, so $X^{*}$ is bounded below by $-1$ and $X^{*}\in L^{\infty}(\mathcal{F}_{s})$.

For the final assertion, we will first show that $\operatorname*{ess\,inf}_{\tau\in\mathcal{T}(s)}f(\tau,X^{*})\leq 0$ $\mathbb{P}$-a.s. For each $n\in\mathbb{N}$, we can again find a sequence $\tau^{n}_{k}$ such that

[TABLE]

By condition (ii), it follows that

[TABLE]

By taking $n\to\infty$, it follows that $\operatorname*{ess\,inf}_{\tau\in\mathcal{T}(s)}f(\tau,X^{*})\leq 0$.

To finish the proof, it suffices to show that for all $\sigma\in\mathcal{T}(s)$, we have $f(\sigma,X^{*})\geq 0$. Fix $\sigma\in\mathcal{T}(s)$. Define $F:=-f(\sigma,X^{*})$ and the event $B:=\left\{F>0\right\}$. Let $\tau_{k}\in\mathcal{T}(s)$ be a sequence such that $f(\tau_{k},X^{*})\searrow\operatorname*{ess\,inf}_{\tau\in\mathcal{T}(s)}f(\tau,X^{*})\leq 0$. We define $Y:=X^{*}\mathbb{I}_{B^{c}}+\left(X^{*}-\frac{F}{2L}\right)\mathbb{I}_{B}\leq X^{*}$ and a sequence $\sigma_{k}=\tau_{k}\mathbb{I}_{B^{c}}+\sigma\mathbb{I}_{B}$. Then

[TABLE]

Hence, $Y\in\mathcal{X}$. By minimality of $X^{*}$, it must follow that $B$ is a $\mathbb{P}$-null set. ∎

Theorem C.7**.**

There exists $\tau^{*}\in\mathcal{T}(s)$ such that $V_{s}(\tau^{*},\lambda)=\operatorname*{ess\,inf}_{\tau\in\mathcal{T}(s)}V_{s}(\tau,\lambda)$.

Proof.

We write $\mathcal{F}^{s}_{n}:=\mathcal{F}_{s+n}$ and define the processes

[TABLE]

where $\tau$ is considered over the space of all stopping times and $C$ is an upper bound given in Assumption 3.1.

Define $\tau^{*}:=\inf\{n\geq 0:Y_{n}=Z_{n}\}$.

By robust representation theorem (Theorem 3.5), we can represent $\mathcal{E}$ as an essential supremum over a familiy of probability measures which satisfy the law of iteration [68, Equation 4]. It then follows Riedel [68, Theorem 3] that if $\tau^{*}<\infty$ $\mathbb{P}$-a.s., then $\tau^{*}$ is an optimal solution.

Hence, to prove the required result, it suffices to prove that $\tau^{*}\in\mathcal{T}(s)$, $Z_{0}=\operatorname*{ess\,inf}_{\tau\in\mathcal{T}(s)}V_{s}(\tau,\lambda)$ and $\tau^{*}<\infty$ $\mathbb{P}$-a.s.

It is clear that we never stop at time [math]. Therefore, $\tau^{*}\in\mathcal{T}(s)$ and $Z_{0}=\operatorname*{ess\,inf}_{\tau\in\mathcal{T}(s)}V_{s}(\tau,\lambda)$.

On an event $\omega$ such that $\lambda(\omega)<C$ and $h(t)(\omega)\to C$, we can find $N(\omega)$ sufficiently large such that $\sum_{t=N+1}^{n}\beta^{t}\left(h(s+t)-\lambda\right)(\omega)>0$ for all $n$. In particular, we have $\tau^{*}(\omega)\leq N(\omega)<\infty$. Therefore, we have $\tau^{*}<\infty$ $\mathbb{P}$-a.s. and thus it must be optimal. ∎

Lemma C.8**.**

Let $(\rho_{n})_{0\leq n\leq L-1}$ be a simple form of $(\tau,p)$. Then the sequence $(\rho_{n})_{0\leq n\leq L-1}$ can be expressed recursively by the following relation. Set $\rho_{0}=p_{0}$ and define

[TABLE]

and where

[TABLE]

Proof.

To see this, we view $\hat{F}^{(m)}_{n}$ as the number of runs (under $(\tau,p)$) of the $m$th bandit before making the $n$th play. We then consider $\pi^{(m)}_{n}$ as the total number of trials in the $m$th bandit required to complete the $\hat{F}^{(m)}_{n}$th run.

If the run is not yet completed before the $n$th play, we continue to play the same machine. (i.e. we define $\rho_{n}=\rho_{n-1}$). If the run is completed in the $(n-1)$th play, we make a new decision in the $n$th play based on the choice sequence $p$. In that case, we have already made $\psi_{n}=\sum_{m=1}^{M}\hat{F}^{(m)}_{n}$ decisions before the $n$th play. By our convention to start at $\rho_{0}$, the decision of the $n$th play is given by $p_{\psi_{n}}$. ∎

Definition C.9**.**

Given a simple form choice sequence $\rho$, we define the decision filtration induced by $\rho$ by

[TABLE]

where $\eta$ is the recording sequence corresponding to $(\mathfrak{1},\rho)$.

Lemma C.10**.**

The sequence of $\sigma$-algebras $\left(\mathcal{G}^{\rho}_{n}\right)$ given in Definition C.9 forms a filtration, i.e. $\mathcal{G}^{\rho}_{n}\subseteq\mathcal{G}^{\rho}_{n+1}$.

Proof.

Suppose that $A\in\mathcal{G}^{\rho}_{n}$, then for $r\in\mathcal{S}$ and $m\in\{0\}\cup\mathcal{M}$,

[TABLE]

Moreover, by definition, $\{\rho_{n}=m\}\in\mathcal{G}^{\rho}_{n}$, it follows that

[TABLE]

Hence, $A\in\mathcal{G}^{\rho}_{n+1}$. ∎

Lemma C.11**.**

With $(\xi^{\rho}_{n})$ as in Remark 23 and $(\mathcal{G}^{\rho}_{n})$ in Definition C.9, $\xi^{\rho}_{n}$ is $\mathcal{G}^{\rho}_{n}$-measurable. In particular, $\mathcal{H}^{\rho}_{n}:=\sigma\left(\xi^{\rho}_{1},...,\xi^{\rho}_{n}\right)\subseteq\mathcal{G}^{\rho}_{n}.$

Proof.

By Lemma C.10, $\rho_{n-1}$ is $\mathcal{G}^{\rho}_{n}$-measurable. Hence, for a fixed $B\in\mathcal{B}$, we have

[TABLE]

Therefore, $\xi^{\rho}_{n}$ is $\mathcal{G}^{\rho}_{n}$-measurable and the last assertion follows from Lemma C.10. ∎

Theorem C.12**.**

With $(\mathcal{H}^{\rho}_{n})_{n\geq 0}$ as in Remark 23 and $(\mathcal{G}^{\rho}_{n})_{n\geq 0}$ in Definition C.9, $\mathcal{G}^{\rho}_{n}=\mathcal{H}^{\rho}_{n}$.

Proof.

We will prove this result by induction. It is clear that $\mathcal{G}^{\rho}_{0}=\mathcal{H}^{\rho}_{0}$. We now assume that $\mathcal{G}^{\rho}_{n-1}=\mathcal{H}^{\rho}_{n-1}$.

By Lemma C.11, it suffices to show that $\mathcal{G}^{\rho}_{n}\subseteq\mathcal{H}^{\rho}_{n}$. Recall from Definition 4.5 that for $n\leq L$, the recording sequence $\eta_{n}$ takes values in

[TABLE]

For a fixed $r\in{\mathcal{S}}_{N}$, we define $\mathcal{P}_{r}$ to be the space of sequences of length $\sum_{m=1}^{M}r^{(m)}$ with values in $\mathcal{M}$ with exactly $r^{(m)}$ replications of $m\in\mathcal{M}$. (e.g. for $\mathcal{M}=\{1,2\}$, $\mathcal{P}_{(3,1)}=\{(1,1,1,2),(1,1,2,1),(1,2,1,1),(2,1,1,1)\}$).

For $A\in\mathcal{G}^{\rho}_{n}$ and $\pi\in\mathcal{P}_{r}$,

[TABLE]

By the Doob–Dynkin lemma, there exists a measurable function $f:\mathbb{R}^{n}\to\mathbb{R}$ such that

[TABLE]

for some measurable function ${f}_{\pi}$ defined by reordering the input of $f$ by $\pi$.

By the inductive hypothesis, $\mathbb{I}_{\left\{\left(\rho_{k}\right)_{0\leq k\leq n-1}=\pi\right\}}$ is $\mathcal{G}^{\rho}_{n-1}$-measurable and thus $\mathcal{H}^{\rho}_{n-1}$-measurable. Hence, the RHS of (C.6) is $\mathcal{H}^{\rho}_{n}$-measurable. Therefore, $\mathbb{I}_{A}=\sum_{r\in{\mathcal{S}}_{N}}\sum_{\pi\in\mathcal{P}_{r}}\mathbb{I}_{A\cap\left\{\left(\rho_{k}\right)_{0\leq k\leq n-1}=\pi\right\}}$ is $\mathcal{H}^{\rho}_{n}$-measurable, i.e. $A\in\mathcal{H}^{\rho}_{n}$, which completes the proof. ∎

Appendix D Numerical Algorithm to Estimate Robust Gittins index

To approximate the value of $\gamma_{k,\beta,T}\left(p,1/\sqrt{n}\right)$ in the setting of Section 5, we proceed following the rough recipe below. (The code used is available on request.)

Fix a grid of values for $\gamma\in G\subseteq[0,1]$.

Set V^{\gamma}_{T}\Big{(}p,\frac{1}{\sqrt{n+T}}\Big{)}=0. 2. 2.

Assume that we know $V^{\gamma}_{t+1}$. We evaluate the backward recursion

[TABLE]

This is done by considering the discrete values of $p$ and using linear interpolation over $[0,1]$. 3. 3.

Using these iterates, determine the initial value function

[TABLE]

By Snell’s envelope argument,

[TABLE] 4. 4.

Repeat step 1-3 for all $\gamma\in G$. 5. 5.

Calculate $\gamma_{k,\beta,T}(p,1/\sqrt{n})$ for a fixed $(p,1/\sqrt{n})$ by

[TABLE] 6. 6.

Repeat the previous step to compute $\gamma_{k,\beta,T}(p,1/\sqrt{n})$ for other values of $(p,1/\sqrt{n})$ and obtain the surface by linear interpolation.