Reinforcement with Fading Memories

TL;DR

This paper analyzes how imperfect memory affects decision-making in stochastic sequential tasks, deriving formulas for steady-state choices and highlighting the importance of the update-to-decay rate ratio.

Contribution

It provides closed-form solutions for the steady-state choice distribution under large memory span and elucidates the critical role of update and decay rates in decision accuracy.

Findings

When update rate exceeds decay rate, the agent nearly always chooses the optimal action.

If decay rate exceeds update rate, choices are proportional to reward rates.

The model offers insights into decision-making with fading memories in stochastic environments.

Abstract

We study the effect of imperfect memory on decision making in the context of a stochastic sequential action-reward problem. An agent chooses a sequence of actions which generate discrete rewards at different rates. She is allowed to make new choices at rate $\beta$, while past rewards disappear from her memory at rate $\mu$. We focus on a family of decision rules where the agent makes a new choice by randomly selecting an action with a probability approximately proportional to the amount of past rewards associated with each action in her memory. We provide closed-form formulae for the agent's steady-state choice distribution in the regime where the memory span is large ($\mu \to 0$), and show that the agent's success critically depends on how quickly she updates her choices relative to the speed of memory decay. If $\beta \gg \mu$, the agent almost always chooses the best action, i.e., the one with the highest reward rate. Conversely, if $\beta \ll \mu$, the agent chooses an action with a probability roughly proportional to its reward rate.

Equations

$$\mathbb{P}(\text{mbox}newchoice=k)=\frac{Q_k(t)\vee \alpha }{{\sum }_{i\in \mathcal{K}}(Q_i(t)\vee \alpha )},k\in \mathcal{K},$$

$$\underset{t\to \infty }{\lim }\mathbb{P}(C(t)=k),k\in \mathcal{K},$$

$$\mu =1/m.$$

$$c_k=\underset{m\to \infty }{\lim }\mathbb{P}(C^m(\infty )=k),k\in \mathcal{K}.$$

$$c_1=$$

$$c_k=$$

$$c_k=1/K,k=1,\dots ,K.$$

$$c_k=\frac{{\lambda }_k\vee {\alpha }_0+(K-1){\alpha }_0}{{\sum }_{i\in \mathcal{K}}\left[{\lambda }_i\vee {\alpha }_0+(K-1){\alpha }_0\right]},k=1,\dots ,K.$$

$${\overline{Q}}^m(t)=\frac{1}{m}{Q}^m(mt),t\in {\mathbb{R}}_{+}.$$

$$q_k=\underset{m\to \infty }{\lim }\mathbb{E}({\overline{Q}}_k^m(\infty )),k\in \mathcal{K}.$$

$$q_1=$$

$$q_k=$$

$$q_k={\lambda }_k/K,k=1,\dots ,K.$$

$$q_k={\lambda }_k\frac{{\lambda }_k\vee {\alpha }_0+(K-1){\alpha }_0}{{\sum }_{i\in \mathcal{K}}\left[{\lambda }_i\vee {\alpha }_0+(K-1){\alpha }_0\right]},k=1,\dots ,K.$$

$$W^m(t)=(Q^m(t),C^m(t)),t\in {\mathbb{R}}_{+}.$$

$$p_k(q)=\frac{q_k\vee {\alpha }_0}{{\sum }_{i\in \mathcal{K}}{q}_i\vee {\alpha }_0},q\in {\mathbb{R}}_{+}^{\mathcal{K}},k\in \mathcal{K}.$$

$${\dot{q}}_k(t)={\lambda }_k{p}_k(q(t))-q_k(t),\forall k\in \mathcal{K},$$

$$\displaystyle q^{I}_{k}=\left\{\begin{array}[]{ll}\lambda_{1}-(K-1)\alpha_{0},&\quad k=1,\\ (\lambda_{k}/{\lambda_{1}})\alpha_{0},&\quad k=2,\ldots,K.\\ \end{array}\right.$$

$$q_k^I=\frac{{\lambda }_k}{K},k=1,\dots ,K.$$

$${\lambda }_k\frac{q_k^I\vee {\alpha }_0}{{\sum }_{i\in \mathcal{K}}{q}_i^I\vee {\alpha }_0}-q_k^I=0,\forall k\in \mathcal{K}.$$

$${\mathcal{K}}^{+}=$$

$$q_k^I\vee {\alpha }_0=q_k^I,k\in {\mathcal{K}}^{+},$$

$$q_k^I\vee {\alpha }_0={\alpha }_0,k\in {\mathcal{K}}^{-}.$$

$$q_1^I={\lambda }_1\frac{{\alpha }_0}{({\sum }_{i\in {\mathcal{K}}^{+}}{q}_i^I)+{\alpha }_0|{\mathcal{K}}^{-}|}={\lambda }_1\frac{{\alpha }_0}{K{\alpha }_0}=\frac{{\lambda }_1}{K}>{\alpha }_0,$$

$${\lambda }_k\frac{q_k^I}{({\sum }_{i\in {\mathcal{K}}^{+}}{q}_i^I)+{\alpha }_0|{\mathcal{K}}^{-}|}=q_k^I,$$

$$\sum _{i\in {\mathcal{K}}^{+}}{q}_i^I={\lambda }_k-{\alpha }_0|{\mathcal{K}}^{-}|.$$

$$q_1^I={\lambda }_1\frac{{\alpha }_0}{({\sum }_{i\in {\mathcal{K}}^{+}}{q}_i^I)+{\alpha }_0|{\mathcal{K}}^{-}|}\stackrel{(a)}{=}{\alpha }_0\frac{{\lambda }_1}{\tilde{\lambda }}\stackrel{(b)}{>}{\alpha }_0,$$

$$q_1^I=\sum _{i\in {\mathcal{K}}^{+}}{q}_i^I={\lambda }_1-{\alpha }_0|{\mathcal{K}}^{-}|={\lambda }_1-(K-1){\alpha }_0.$$

$$q_k^I={\lambda }_k\frac{{\alpha }_0}{({\sum }_{i\in {\mathcal{K}}^{+}}{q}_i^I)+{\alpha }_0|{\mathcal{K}}^{-}|}={\alpha }_0\frac{{\lambda }_k}{{\lambda }_1},$$

$$\left(\sum _{i\in {\mathcal{K}}^{+}}{q}_i^I\right)+{\alpha }_0|{\mathcal{K}}^{-}|=q_1^I+(K-1){\alpha }_0=[{\lambda }_1-(K-1){\alpha }_0]+(K-1){\alpha }_0={\lambda }_1.$$

$$q_1^I={\lambda }_1-(K-1){\alpha }_0\le {\lambda }_1-(1-1/K){\lambda }_1=\frac{{\lambda }_1}{K}\le {\alpha }_0,$$

$$q_k^I={\lambda }_k\frac{{\alpha }_0}{{\alpha }_{0}K}={\lambda }_{k}K,k=1,\dots ,K.$$

Full text

We provide closed-form formulae for the agent’s steady-state choice distribution in the regime where the memory span is large ($\mu\to 0$), and show that the agent’s success critically depends on how quickly she updates her choices relative to the speed of memory decay. If $\beta\gg\mu$, the agent almost always chooses the best action, i.e., the one with the highest reward rate. Conversely, if $\beta\ll\mu$, the agent chooses an action with a probability roughly proportional to its reward rate.111September 2017; revised: September 2019. An extended abstract of this paper appeared in ACM SIGMETRICS, 2018.

Keywords: memory, stochastic model, $M/M/\infty$ queue, Markov process, fluid model, reinforcement.

1 Introduction

Memories serve as a crucial link between the past and future in dynamic decision-making problems, allowing subsequent decisions to draw on earlier experiences. However, they are hardly perfect in reality: humans routinely rely on faulty memories when making choices, and a firm’s time-varying states, such as an active user base that is prone to unpredictable attrition, could sway its strategic focus. This raises the question: to what extent can we make good decisions despite having an imperfect memory? In this paper, we investigate this question in the context of a stochastic action-reward model. While the subject can be approached from many different angles, we will focus on understanding the interplay between the rate of memory decay and the rate of decision updates; the former a measure of the quality of memory, and the latter a proxy of the decision maker’s temporal adaptivity. Our main results demonstrate that the relative magnitude between the two rates can have profound performance implications.

Let us begin with an informal description of our model (Figure 1). An agent operating in continuous time makes choices among a finite menu of actions. When action $k$ is chosen, the agent accrues discrete rewards according to a Poisson process with rate $\lambda_{k}$, while a unit of reward “expires” and disappears from her memory after a random amount of time that is exponentially distributed with rate $\mu$. Opportunities for the agent to make new choices arise according to a Poisson process with rate $\beta$. We will restrict our attention to a simple family of decision rules that the agent may use, referred to as the reward matching rule, which reinforces actions with more positive past stimuli in proportional manner: she randomly samples an action such that the probability of choosing action $k$ is proportional to $\max\{Q_{k},\alpha\}$, where $Q_{k}$ is the total units of rewards accrued through action $k$ that have yet to expire, and $\alpha>0$ an exploration parameter representing the agent’s minimal willingness to experiment with any action. If $\alpha$ is set to 0, then the reward matching rule corresponds to the celebrated Luce’s linear probabilistic choice model (Luce, 1959, Erev and Roth, 1998), which has been extensively studied in behavioral economics (Erev and Roth, 1998, Beggs, 2005) and evolutionary biology (Harley, 1981). We are interested in the probability that the agent chooses a particular action in steady state. We next examine two illustrative examples, which also serve as motivating applications for the more stylized model we study.

Example 1 - Consumer Choice Modeling. We may think of the agent as a consumer, the actions as products or service providers, and the opportunities to choose new actions as the times when the consumer is allowed to re-select her membership or subscription. The rewards act as positive experiences or impressions resulting from using a service, and the strength of each impression in the agent’s memory diminishes as time progresses. When an opportunity to renew the subscription arises, the agent chooses a new service offering in a manner that is biased towards the ones associated with more recallable impressions.

Example 2 - Dynamic Product Offering under Customer Attrition. In this example, we slightly refine the model by letting the departure rates of rewards vary across actions. Consider a firm who operates a number of service contracts (actions), e.g., mobile phone plans, and needs to decide which contract to offer new customers (rewards) in a series of promotion periods, where one contract is offered for each period. Customers arrive to the system and are subscribe to the service contract of the corresponding promotion period. Those who subscribe to contract type $k$ remain an active customer for an exponentially distributed amount of time with rate $\mu_{k}$, before departing from the system. The departure rate $\mu_{k}$ reflects some underlying qualities of the contract $k$. The reward matching rule corresponds to the service provider’s choosing, at the end of each promotion period, a new contract with probabilities roughly proportional to the number of active customers associated with each contract.

Motivation for studying the reward matching rule.

An implied objective of our model is that the agent would like to maximize the steady-state total amount of rewards in system. If all the parameters were known a priori, it is not difficult to see that this objective could be trivially achieved by choosing at all times the best action, i.e., one with the highest reward rate $\lambda_{k}$ (Example 1) or the highest weighted reward rate, $\lambda_{k}/\mu_{k}$ (Example 2). The reward matching rule, therefore, can be viewed as an intuitive heuristic for approximately solving this optimization problem in the absence of the knowledge of the parameters. While the rule is by no means the only plausible heuristic for this purpose, we believe that it has a number of advantages which merit a theoretical investigation such as the one carried out in this work:

Firstly, Luce’s rule, after which the reward matching rule is modeled, has a long history as a fundamental building block in psychology and behavioral economics (see Section 3 for more details). The reward matching rule hence serves a natural starting point for understanding other, potentially more complex, cognitive models. 2. 2.

Secondly, the reward matching rule is useful not only for modeling human choice behavior. It could also be a policy deliberately chosen by a decision maker, such as in Example 2. While a conscious decision maker could in principle adopt more sophisticated decision rules or even use external information storage to mitigate memory decay, the reward matching rule offers a simple and intuitive rule of thumb that can be substantially easier for a practitioner to understand and implement, and is applicable in settings where external information storage is not possible. 3. 3.

Last but not least, our results show that, under a large number of parameter values, the reward matching rule is in fact near optimal and induces a steady-state choice probabilities that heavily concentrate on the best action. Along with the simplicity and conceptual appeal, the near-optimality therefore further justifies studying the reward-matching rule.

Preview of main results.

Our main results provide simple, closed-form formulae for the distribution of the agent’s choice in steady state, in the regime where the agent’s memory span is large ($\mu\to 0$). Using these formulae, we show that the agent’s success critically depends on how fast her choices are updated ($\beta$) relative to the rate of memory decay ($\mu$), when $\alpha$ is relatively small compared to the overall memory size. In particular, if $\beta\gg\mu$,222The notation $\beta\gg\mu$ represents $\lim_{\mu\to 0}{\beta}/{\mu}=0$. the agent nearly always chooses the best action, i.e., the one with the highest reward rate $\lambda_{k}$. In contrast, if $\beta\ll\mu$, the agent splits her attention more smoothly among all actions, with frequencies roughly proportional to the respective $\lambda_{k}$’s.

1.1 The Model

We now describe our model more formally, as is depicted in Figure 1. The system consists of an agent operating in continuous time, who makes choices by selecting from a set of $K$ actions, $\mathcal{K}=\{1,\ldots,K\}$. We denote by $C(t)\in\mathcal{K}$ the action chosen by the agent at time $t$, and refer to $\{C(t)\}_{t\in\mathbb{R}_{+}}$ as the choice process. When the $C(t)=k$ for some $k\in\mathcal{K}$, the agent accrues discrete rewards according to a Poisson process with rate $\lambda_{k}>0$, where each reward can be treated as a token of unit size. We refer to $\lambda_{k}$ as the reward rate of action $k$. We assume that there exists one reward rate that strictly dominates the rest, and, without loss of generality, we rank them in a decreasing order, so that $\lambda_{1}>\lambda_{2}\geq\ldots\geq\lambda_{K}>0$.

The agent’s fading memory is modeled by the “expiration” of past rewards. Each reward is associated with a lifespan, an exponential random variable with rate $\mu>0$ drawn independently from all other aspects of the system, and a reward departs permanently after staying in the system for a time duration equal to its lifespan. We refer to $\mu$ as the rate of memory decay. For simplicity of notation, the majority of our exposition and proofs will focus on the case where $\mu$ is uniform across all actions, but our main results extend to the case with action-dependent memory decay rates in a straightforward manner; such a refinement will be discussed in Section 8.3.

For $k\in\mathcal{K}$, we denote by $Q_{k}(t)$ the total units of rewards accrued from choosing action $k$ that have yet to depart by time $t$, and we refer to $\{Q(t)\}_{t\in\mathbb{R}_{+}}$ as the recallable reward process. Note that at time $t$ the recallable rewards associated with action $k$ depart at an aggregate rate of $\mu Q_{k}(t)$, and arrive at rate $\lambda_{k}$, if $C(t)=k$, or [math], otherwise. We assume that the agent’s initial recallable rewards at time [math] is a bounded vector in $\mathbb{Z}_{+}^{\mathcal{K}}$.

The agent has the opportunity to make a new choice at a set of update points scattered across time. The update points are distributed according to an exogenous Poisson process with rate $\beta$, which we refer to as the update rate, so that the time between two adjacent update points is an exponentially distributed random variable with mean $\beta^{-1}$, independent from all other aspects of the system. Depending on the application context, $\beta$ could be interpreted as the level of “activeness” or “savvyness,” as it modulates the frequency at which the agent updates her decisions according to the present state of her recallable-memories (cf. Examples 1 and 2 in the Introduction).

How does the agent make a new choice when an update point arises? In this work, we will focus on a family of choice heuristics referred to as the reward matching rules. We assume that at time $t$ the agent makes a new choice by sampling from the distribution

[TABLE]

where $x\vee y\stackrel{{\scriptstyle\triangle}}{{=}}\max\{x,y\}$. Here, $\alpha$ is a positive constant referred to as the exploration parameter, which captures the agent’s willingness to experiment with an action even if there is currently little or no recallable reward attached to it. The justification and background behind the choice of this decision rule will be further elaborated on in Section 3. The parameter $\alpha$ can also have an effect on the trade-off between the system’s steady-state versus transient performance; the reader is referred to Section 8.1.1 for a discussion.

1.2 Performance Metrics and Scaling Regime

The main goal of this paper is to study the agent’s long-run behavior in the model described above, and a key quantity of interest is the steady-state distribution of the choice process, $C(\cdot)$, i.e.,

[TABLE]

Unfortunately, the choice process is non-Markovian as its transitions are influenced by the recallable reward process $Q(\cdot)$ in a non-trivial way, and this makes it difficult to obtain explicit expressions for its steady-state distribution. To circumvent this challenge, we will focus on the regime where the average lifespan of a reward, $\mu^{-1}$, tends to infinity, as follows. Fix the action set, $\mathcal{K}$, and the reward rates, $\{\lambda_{k}\}_{k\in\mathcal{K}}$. We consider a sequence of systems, indexed by $m\in\mathbb{N}$, where the average lifespan of a reward in the $m$th system is equal to $m$, i.e.,

[TABLE]

We will denote by $\alpha_{m}$, $\beta_{m}$, $Q^{m}(\cdot)$, and $C^{m}(\cdot)$ the corresponding quantities in the $m$th system. Finally, as the rewards’ average lifespan, $m$, tends to infinity, the total amount of recallable rewards in the system also scales as $\Theta(m)$.333To see this, note that if the choice process were to stay in action $1$ all the time, then $Q^{m}_{1}(\cdot)$ would evolve according to an $M/M/\infty$ queue with arrival rate $\lambda_{1}$ and departure rate $m^{-1}$, leading to a steady-state expected total recallable reward of $\lambda_{1}m$. For this reason, we will scale the exploration parameter $\alpha_{m}$ linearly with respect to $m$, by setting $\alpha_{m}=\alpha_{0}*m$, where $\alpha_{0}$ is a fixed parameter.

Remark. It is natural to ask whether by taking a sequence of systems where the memory decay rate approaches zero, we will end up with the trivial case where there is no memory loss at all ($\mu=0$), which would have been at odds with our goal of studying imperfect memories. This not the case. Because we always look at the steady state of the system by taking the limit as $t\to\infty$ for every system in the sequence, the limiting regime is non-trivial and is not equivalent to simply setting $\mu=0$, as is evidenced by the existence of two distinct limiting steady-state distributions in our main theorems (This point is further elaborated on in Appendix A.) The limit of $\mu\to 0$ is intended as an approximation for a pre-limit system where $\mu$ is positive but small. Numerical results in Figure 4 and 5 suggest that our theorems provide a reasonably good approximation for $m$ as small as $200$.

1.3 Notation

For $x,y\in\mathbb{R}$, $x\vee y$ and $x\wedge y$ denote $\max\{x,y\}$ and $\min\{x,y\}$, respectively. For a vector $x\in\mathbb{R}_{+}^{K}$, $\|x\|$ denotes the $L_{1}$ norm of $x$, $\|x\|=\sum_{i=1}^{K}|x_{i}|$.

We use the notation $X(\cdot)$ and $X[\cdot]$ as a shorthand for processes $\{X(t)\}_{t\in\mathbb{R}_{+}}$ and $\{X[n]\}_{n\in\mathbb{N}}$, respectively. For two random variables $X$ and $Y$ taking values in $\mathbb{R}$, we use $X\stackrel{{\scriptstyle d}}{{=}}Y$ to mean that $X$ and $Y$ have the same distribution. The expression $X\preceq Y$ means $X$ is stochastically dominated by $Y$, i.e., $\mathbb{P}(X\geq x)\leq\mathbb{P}(Y\geq x)$ for all $x\in\mathbb{R}$. For an event $\mathcal{A}$, we use $X|\mathcal{A}$ to denote the random variable with the cumulative distribution function \mathbb{P}(X\leq x\,\big{|}\,\mathcal{A}), $x\in\mathbb{R}$. For a stochastic process, $\{X(t)\}_{t\in\mathbb{R}_{+}}$, we denote by $X(\infty)$ the random variable distributed according to the stationary distribution of $\{X(t)\}_{t\in\mathbb{R}_{+}}$, if it exists.

For two functions $f$ and $g$, $f\gg g$ means that $\lim_{x\to\infty}\frac{f(x)}{g(x)}=\infty$; $f\ll g$ is defined analogously. When such a limiting notation is involved, the limit is assumed to be as the argument tends to $\infty$, unless explicitly specified otherwise.

2 Main Results

We now present our main results, which provide exact expressions for the steady-state distribution of the choice process, as well as the expected value of the (scaled) recallable reward process, in the limit as $m\to\infty$. For the remainder of the paper, we will assume that the update rate $\beta_{m}$ satisfies either $\beta_{m}\gg 1/m$ or $\beta_{m}\ll 1/m$, as $m\to\infty$.

Theorem 2.1** (Steady-State Choice Probabilities)**

Fix $\alpha_{0}>0$. The following limits exist

[TABLE]

Furthermore, their expressions depend on the scaling of $\beta_{m}$, as follows.

(Memory-abundant regime)* Suppose that $\beta_{m}\gg 1/m$, as $m\to\infty$. If $\alpha_{0}\leq\lambda_{1}/K$, then*

[TABLE]

If $\alpha_{0}>\lambda_{1}/K$, then

[TABLE] 2. 2.

(Memory-deficient regime)* Suppose that $\beta_{m}\ll 1/m$, as $m\to\infty$. Then,*

[TABLE]

The next theorem characterizes the steady-state expected value of the recallable reward process, $Q^{m}(\cdot)$. For $m\in\mathbb{N}$, define the scaled recallable reward process:

[TABLE]

Theorem 2.2** (Steady-State Recallable Rewards)**

Fix $\alpha_{0}>0$. For all $m\in\mathbb{N}$, the scaled recallable reward process, $\{\overline{Q}^{m}(t)\}_{t\in\mathbb{R}_{+}}$, admits a unique steady-state distribution, and the following limits exist:

[TABLE]

Furthermore, their expressions depend on the scaling of $\beta_{m}$, as follows.

(Memory-abundant regime)* Suppose that $\beta_{m}\gg 1/m$, as $m\to\infty$. If $\alpha_{0}\leq\lambda_{1}/K$, then*

[TABLE]

If $\alpha_{0}>\lambda_{1}/K$, then

[TABLE] 2. 2.

(Memory-deficient regime)* Suppose that $\beta_{m}\ll 1/m$, as $m\to\infty$. Then,*

[TABLE]

2.1 Implications of the Theorems

A prominent feature of Theorems 2.1 and 2.2 is that the results critically depend on how quickly the update rate $\beta_{m}$ scales relative to the rate of memory decay $1/m$, but are otherwise insensitive to the exact expression of $\beta_{m}$. This leads to a natural dichotomy of the system dynamics into what we called the memory-abundant and memory-deficient regimes, and we discuss below several notable properties of the two regimes.

Near optimality and winner-takes-all.

Suppose that the constant $\alpha_{0}$ in the exploration parameter is substantially smaller than $\lambda_{1}/K$. Then, our results show that the best action attracts nearly all of the agent’s attention, if $\beta_{m}\ll 1/m$ (memory-abundant regime). Specifically, in the limit as $m\to\infty$, the agent chooses action $1$ with a probability of $(1-\frac{K-1}{\lambda_{1}}\alpha_{0})$, which is almost $1$ when $\alpha_{0}$ is very small compared to $\lambda_{1}/K$. Recall from Section 1 that the optimal strategy for the agent, should she know all parameters of the problem, would be to choose action $1$ at all times. Therefore, in this regime, the reward matching rule is able to deliver near optimal performance.

Moreover, the system exhibits an interesting winner-takes-all phenomenon, whereby the degree to which the agent focuses on the best action is independent of the reward rates of all other actions. For instance, if we were to increase the reward rate of the second-best action, $\lambda_{2}$, the probability that the agent chooses action $1$ would stay unchanged (and close to 1), and that of choosing action $2$ would still be close to zero no matter how close $\lambda_{2}$ is to $\lambda_{1}$. One consequence of this effect is that, if we imagine the actions as being “competitive” service providers vying for a consumer’s attention, then the winner-takes-all phenomenon can be interpreted as a form of extreme competition among the providers, where the best attracts nearly all of the consumer’s budget, even if its superiority compared to the second best is not significant.

Reward-rate-proportional allocation.

In contrast to the memory-abundant regime, an update rate that scales substantially slower than the memory decay rate will lead to an allocation of choices that is much smoother as a function of the reward rates (Figure 3 provides a graphical comparison of the qualitative difference between the agent’s behavior under the two regimes). In this case, the probability that the agent chooses action $k$ is proportional to $[\lambda_{k}\vee\alpha_{0}+(K-1)\alpha_{0}]$, which, when $\alpha_{0}$ is small, is essentially equal to the reward rate, $\lambda_{k}$. The agent’s behavior thus exhibits a certain weak reinforcement, where the better actions do attract more attention but only proportional to their respective reward rates, and the above-mentioned winner-takes-all effect disappears.

The intuition behind this reward-rate-proportional allocation can be roughly explained as follows. When the update rate is slow, by the time a new choice is to be made the agent would have forgotten the rewards associated with all actions other than her most recent choice. Nevertheless, she still exhibits a mild preference over actions with higher reward rates in steady state, because she does retain a good memory of the rewards of her most recent choice, $C(t)$, which can be shown to be proportional to $\lambda_{C(t)}$. The reward rate $\lambda_{C(t)}$ dictates how likely the agent will make the same choices, which in turn leads to the reward rates’ approximately linear appearance in the steady-state distribution of the choice process.

Rapid updates lead to complete oblivion.

The theorems also show that, perhaps surprisingly, a slower update rate is not always bad. While it may be tempting to conclude that faster update rates always lead to better outcomes for the agent, this turns out to be true only when the exploration parameter is relatively small. In fact, as soon as $\alpha_{0}$ grows beyond $\lambda_{1}/K$, in the memory-abundant regime, the agent becomes completely oblivious and chooses actions uniformly at random (Eq. (6)). In contrast, the agent is always biased towards the best action in the memory-deficient regime, albeit mildly, as long as $\alpha_{0}<\lambda_{1}$ (beyond which point the exploration parameter is so large that it trivially overwhelms even the best action). The true culprit behind the complete oblivion is a lack of patience: when the update rate is fast and the exploration parameter high, the agent tends to switch her choices rapidly, before she is able to collect enough rewards to “learn” the reward rate of any individual action. Consequently, in the long run, not a single component of the recallable reward, not even the best action, can distinguish itself by consistently rising above the exploration parameter, and hence complete oblivion becomes the only possible outcome.

Remark 2.3

Note that Theorems 2.1 and 2.2 have left out the regime where $\beta$ and $\mu$ scale at the same rate, i.e., when $\beta=\Theta(\mu)$ as $\mu\to 0$. This has turned out to be a difficult regime to tackle, and an intuitive explanation for this difficulty is as follows. Our current proof technique exploits the fact that one of the two processes in $Q(\cdot)$ and $C(\cdot)$ becomes asymptotic Markovian property as $\mu\to 0$, which significantly simplifies the dynamics (see Section 4). Unfortunately, this is no longer the case when $\beta=\Theta(\mu)$, as neither of the two processes $Q(\cdot)$ and $C(\cdot)$ becomes asymptotically Markovian by itself, and the stochastic dynamics of the system remain complex despite taking the limit of $\mu\to 0$. We may illustrate this phenomenon by the following back-of-the-envelope calculation: the probability that any individual unit of reward will depart from the system between two consecutive update points is on the order of $\beta\cdot\mu^{-1}$. When $\beta=\Theta(\mu)$, the number of reward departures between two update points forms a constant fraction of existing rewards and the same is true for the number of new rewards arriving during this period. Therefore, the changes in the recallable-reward process $Q(\cdot)$ remain highly variable from one update point to the next, and they depend both on the process $Q(\cdot)$ itself and the choice process, $C(\cdot)$. In contrast, in the memory-abundant or memory-deficient regimes, the profile of the recallable rewards either changes very little between two update points (memory-abundant regime), or substantially but in a predictable manner (memory-deficient regime).

2.2 Numerical Results

While Theorems 2.1 and 2.2 only apply in the limit where the average reward lifespan tends to infinity, the simulation results in Figures 4 and 5 show that our theoretical results also provide fairly accurate predictions, quantitatively and qualitatively, in systems with a moderate, finite lifespan. In Figure 4, we fix $m$, while varying the update rate, $\beta$, and we observe that concentration on the best action intensifies as $\beta$ increases. Figure 5 shows example sample paths of the scaled recallable reward processes, $\overline{Q}^{m}(\cdot)$. One can discern without difficulty that, even for a modest $m$, the process in the memory-abundant regime (plots $(a)$ through $(c)$) stays close to a set of smooth “fluid” trajectories, whose invariant state (the flat portion of the trajectories) coincide with the limiting value of $\mathbb{E}(\overline{Q}^{m}(\infty))$ predicted in Eq. (10), while in the memory-deficient regime the process changes more abruptly over time. As will become clear in the sequel, this qualitative discrepancy is a direct consequence of the overall system dynamics being dominated by the evolution of either $\overline{Q}^{m}(\cdot)$ or $C^{m}(\cdot)$, depending on the scaling of the update rate $\beta_{m}$, a key fact that we will exploit in our proof of the main theorems.

3 Related Literature

The assumption of a reward’s lifespan being exponentially distributed is modeled after the exponential memory decay theory of Ebbinghaus (Ebbinghaus, (1964)), which posits that the recall probability of a past event decays exponentially as time passes. The reward matching decision rule (Eq. (1)) can be viewed as a generalization of Luce’s linear probabilistic choice rule (Luce, (1959)), originally created to model empirical observations where humans or animal subjects make choices with probabilities proportional to associated amount of past rewards or stimuli. Luce’s rule has been studied extensively in a variety of disciplines, ranging across cognitive science (Herrnstein, (1970)), behavioral economics (Erev and Roth, (1998)), and evolutionary biology (Harley, (1981)); more recently, in a different context, a similar proportional-sampling inference algorithm highly related to the Luce’s rule has been used in establishing the sample complexity of Bayesian active learning with privacy constraints (Xu, (2018)). Noteably, the celebrated work of Erev and Roth, (1998) demonstrates that a discrete-time version of Luce’s rule can be a powerful reinforcement learning model for explaining a player’s behavior in sequential games, and subsequent theoretical analysis showed that the empirical frequency of the best action under Luce’s rule converges to one in single player games (Rustichini, (1999), Beggs, (2005), and more recently, Mertikopoulos and Sandholm, (2016)). These results, however, assume the agent has perfect memory, while models involving memory decay have received relatively little attention. Beggs, (2005) analyzes a variation of the model involving “forgetting” as a means to speed up convergence to mixed equilibria, with a crucial feature that the forgetting be performed in a noiseless fashion by weighing distant past experience with a deterministic discount factor. As such, among other differences, the stochastic memory decay model that we adopt exhibits fundamentally different dynamics than the deterministic discount model; for instance, under our model there is a non-negligible probability that all past rewards could be entirely erased, while the deterministic version never truly forgets the past, no matter how distant.

The impact of limited memory to sequential decision-making has been examined in the statistics literature dating back to the seminal work of Cover and Hellman, (1970), Hellman and Cover, (1970), which proposed algorithms for solving multi-arm bandit and sequential hypothesis testing problems when the decision maker has access to only a finite number of bits of memory. In the queueing theory literature, Mitzenmacher et al., (2002) and Gamarnik et al., (2016) study load balancing problems with space-bounded routing algorithms. In contrast to the present paper, the memories in these models are assumed to be perfect and immune to random erasures.

The dynamic induced in our model is related to the celebrated Pólya’s urn scheme (cf. Pemantle, (2007)). In its basic form, one ball is drawn uniformly at random from an urn containing balls of different colors. The chosen ball is then returned to the urn along with a new ball of the same color, and the procedure repeats. Since the probability that a certain color being chosen is proportional to the number of balls of that color present in the urn, by viewing the balls as rewards and colors as actions, our model can be thought of as the urn scheme with the modifications where (a) each ball disappears from the urn after some random amount of time, corresponding to memory loss, and (b) the number of new balls added is random, whose distribution depends on the color, corresponding to the variations in the reward rates. Note that in Pólya’s urn scheme the total number of balls in the urn eventually tends to infinity as the number of draws increases, while in our model the recallable rewards always stay a finite random variable as a result of memory loss. As such, our model can be viewed as a stationary variation of the urn scheme, which admits a very different dynamic.

On the technical end, our analysis of the memory-abundant regime relies on a certain fluid model to approximate the evolution of the original, stochastic recallable reward process. While the use of fluid models is a popular technique in the literature of queueing networks (cf. Kurtz, 1970, Bramson, 1998, Tsitsiklis and Xu, 2012, Massoulié and Xu, 2018), our work departs from conventional fluid models in a notable way: the recallable reward process is not Markovian, and the analysis hence must take into account the dynamics of the auxiliary choice process. It is worth noting that there has been a large-deviation theory developed for obtaining fluid-like approximation results where the transition rates of a continuous-time jump process are modulated by a finite-state auxiliary chain (cf. Theorem 1 of Mitzenmacher et al., (2002), which is based on Theorem 8.15 of Shwartz and Weiss, (1995)). Unfortunately, these results do not apply in our model for two reasons. First, they require the transition rate of the jump process of interest to be bounded, a condition not met here due to the aggregate departure rate of the rewards being unbounded over the state space. Second, the existing results typically have one scaling parameter, $m$, whereby time is sped up by a factor of $m$, and space scaled down by a factor of $1/m$. We use a similar scaling for the recallable reward process, $Q^{m}(\cdot)$, while the update rate $\beta_{m}$ serves as a second scaling parameter that is not captured by the model in Mitzenmacher et al., (2002), Shwartz and Weiss, (1995). The addition of $\beta_{m}$ is non-trivial: in fact, if $\beta_{m}$ is too small, the fluid approximation does not hold, and a different (memory-deficient) regime arises. The dynamics of our model in the memory-abundant regime are also related to the so-called averaging principle in establishing fluid approximation for Markov processes, where the evolution of a Markov process at a slow time scale is influenced by average behavior of a modulating process at a faster time scale (cf. Perry and Whitt, (2011), Mandelbaum et al., (1998), Evdokimova et al., (2018), Coutin et al., (2010); see also Darling and Norris, (2008) for an overview.) However, in these models the underlying slow process is Markovian (e.g., queue lengths) and the modulating process a function of its value (e.g., the differences in queue lengths in Perry and Whitt, (2011)). In contrast, the recallable reward process in our model is not Markovian due to the existence of the choice process. The non-Markovian nature of our model means that certain martingale properties typically used to establish convergence do not directly follow from existing results. As a result of these aforementioned differences, we will develop our fluid approximation results starting from first principles using arguments developed by Kurtz, (1978), involving elementary properties of continuous-time martingales and certain quasi-fluid processes.

4 Proof Overview

The remainder of the paper is devoted to the proof of Theorem 2.2; the proof of Theorem 2.1 is a straightforward extension of Theorem 2.2 and will be given in Section 7. We discuss in this section some high-level ideas before delving into the details. Our proof consists of two main components corresponding to the memory-abundant and memory-deficient regimes, respectively. The system dynamics turn out to be quite different depending on the scaling of $\beta_{m}$, and as a result, so are our proof techniques. Fix $m\in\mathbb{N}$, and define the joint reward-choice process

[TABLE]

It is not difficult to check that the joint process $\{W^{m}(t)\}_{t\in\mathbb{R}_{+}}$ is Markovian, while, for any fixed $m$, the component processes $\{Q^{m}(t))\}_{t\in\mathbb{R}_{+}}$ and $\{C^{m}(t)\}_{t\in\mathbb{R}_{+}}$ depend on one another and are not individually Markovian (see Figure 2). However, a key observation is that one of the two component processes becomes asymptotically Markovian as $m\to\infty$:

(Memory-abundant) If $\beta_{m}\gg 1/m$ as $m\to\infty$, a large number of choice updates takes place before substantial change occurs in the recallable reward process. As a result, the recallable reward process, $\{Q^{m}(t))\}_{t\in\mathbb{R}_{+}}$ , becomes a sufficient Markov representation of the system dynamics, as $m\to\infty$. 2. 2.

(Memory-deficient) If $\beta_{m}\ll 1/m$ as $m\to\infty$, the interval between two consecutive update points is large. By the time a new choice is to be made, the agent will have “forgotten” nearly all of the rewards associated with the actions other than the most current choice, and the rewards associated with the current choice will have become quite predictable. Here, the choice process $\{C^{m}(t))\}_{t\in\mathbb{R}_{+}}$ becomes asymptotically Markovian as $m\to\infty$.

Our proofs for the two regimes will be tailored to the dichotomy outlined above.

For the memory-abundant regime, our analysis focuses on characterizing the scaled recallable reward process, $\overline{Q}^{m}(\cdot)$, and uses a certain fluid model to show that, as $m\to\infty$, the evolution of $\overline{Q}^{m}(\cdot)$ is well approximated by the solution to a system of ordinary differential equations (ODE), and its stationary distribution converges to the unique invariant state of the ODEs. This portion of the proof will be presented in Section 5. 2. 2.

For the memory-deficient regime, we turn instead to the choice process, $C^{m}(\cdot)$, and use its asymptotic Markovian property to obtain an explicit formula for its steady-state distribution, in the limit as $m\to\infty$. We then leverage this formula to calculate the expected steady-state recallable reward using the stationarity properties of the continuous-time processes. The proof for this portion of the theorem is presented in Section 6.

5 The Memory-abundant Regime

We establish the first claim of Theorem 2.2 in this section, and we will do so by actually proving a stronger statement, that the steady-state distribution of $\overline{Q}^{m}_{k}(\cdot)$ converges in $L^{1}$ to the corresponding expressions in Eqs. (10) and (11), as $m\to\infty$. As was alluded to earlier, our main tool relies on certain fluid solutions, defined as the solutions to a set of ordinary differential equations, whose dynamics will be shown to closely approximate that of $\overline{Q}^{m}(\cdot)$ when $m$ is large. We begin by introducing the fluid solutions and some of their basic properties.

5.1 Fluid Solutions and Their Basic Properties

Define the function $p:\mathbb{R}_{+}^{\mathcal{K}}\to\mathbb{R}_{+}^{\mathcal{K}}$, where

[TABLE]

Definition 5.1** (Fluid Solutions)**

Fix $q^{0}\in\mathbb{R}_{+}^{\mathcal{K}}$. A continuous function, $q:\mathbb{R}_{+}\to\mathbb{R}_{+}^{\mathcal{K}}$, is called a fluid solution with initial condition $q^{0}$, if it satisfies the following:

$q(0)=q^{0}$; 2. 2.

for almost all $t\in\mathbb{R}_{+}$, the function is differentiable along all coordinates, with

[TABLE]

where $p(\cdot)$ is defined in Eq. (14).

We say that $q^{I}\in\mathbb{R}_{+}^{K}$ is an invariant state of the fluid solutions, if by setting $q^{0}=q^{I}$, we have that $q(t)=q^{I}$ for all $t\in\mathbb{R}_{+}$.

The first term on the right-hand side of Eq. (15) corresponds to the instantaneous arrival rate of the rewards, where $p_{k}(q(t))$ is the probability that action $k$ is chosen given the current states of recallable rewards, and $\lambda_{k}$ the reward rate for that action. The second term corresponds to the instantaneous departure rate of rewards, which is proportional to the amount of recallable rewards associated with action $k$. The following lemma states some basic properties of the fluid solutions. The proof is given in Appendix D.1.

Lemma 5.2

For all $q^{0}\in\mathbb{R}_{+}^{K}$, there exists a unique fluid solution $\{q(q^{0},t)\}_{t\in\mathbb{R}_{+}}$ with initial condition $q^{0}$. Furthermore, for all $t\in\mathbb{R}_{+}$, $q(q^{0},t)$ is a continuous function with respect to $q^{0}$.

The next result states that the fluid solutions admit a unique invariant state. Note that the expressions of the invariant state coincide with those in Eqs. (10) and (11) in Theorem 2.2.

Theorem 5.3

Fix $\alpha_{0}>0$. The fluid solutions admit a unique invariant state, $q^{I}$, whose expressions depend on the value of $\alpha_{0}$, as follows.

Suppose $\alpha_{0}\in(0,\lambda_{1}/K]$. Then,

[TABLE] 2. 2.

Suppose $\alpha_{0}>\lambda_{1}/K$. Then,

[TABLE]

*Proof. * It can be easily verified that the expressions in Eqs. (18) and (19) are valid invariant states in their respective regimes of $\alpha_{0}$ values. We next show that they are indeed the unique invariant states in both cases. Let $q^{I}$ be an invariant state of the fluid solutions. Then, $q^{I}$ must satisfy:

[TABLE]

Define a partition of $\mathcal{K}$ into $\mathcal{K}^{+}$ and $\mathcal{K}^{-}$, where:

[TABLE]

Note that by definition,

[TABLE]

Suppose that $\alpha_{0}\in(0,\lambda_{1}/K]$. We next show that in this case $\mathcal{K}^{+}$ is the singleton set, $\{1\}$. First, suppose that $\mathcal{K}^{+}$ is empty, that is, $q^{I}_{k}\leq\alpha_{0}$ for all $k\in\mathcal{K}$. We have that

[TABLE]

where the last inequality follows from the assumption that $\alpha_{0}<\lambda_{1}/K$. This leads to a contradiction, implying that $\mathcal{K}^{+}\neq\emptyset$. Fix $k\in\mathcal{K}^{+}$. Combining Eqs. (20), (22) and (23), we have that

[TABLE]

which, after rearrangement, can be written as

[TABLE]

Note that in Eq. (26) only the term $\lambda_{k}$ on the right-hand depends on $k$. This implies that, if $\mathcal{K}^{+}\neq\emptyset$, then all coordinates in $\mathcal{K}^{+}$ must have the same arrival rate of rewards. Thus, we can assume that there exists $\tilde{\lambda}\in\mathbb{R}_{+}$, such that $\lambda_{k}=\tilde{\lambda}$, for all $k\in\mathcal{K}^{+}$, and it suffices to show that $\tilde{\lambda}=\lambda_{1}$.

For the sake of contradiction, suppose that $\tilde{\lambda}\neq\lambda_{1}$, and hence $\lambda_{1}>\tilde{\lambda}$. This implies that $1\notin\mathcal{K}^{+}$, and hence $q^{I}_{1}\leq\alpha_{0}$. Invoking Eq. (20) again, we have that

[TABLE]

where step $(a)$ follows from Eq. (26), and $(b)$ from the fact that $\lambda_{1}>\tilde{\lambda}$. Clearly, Eq. (27) contradicts with the fact that $q^{I}_{1}\leq\alpha_{0}$. We thus conclude that $\tilde{\lambda}=\lambda_{1}$, $\mathcal{K}^{+}=\{1\}$, and $\mathcal{K}^{-}=\{2,\ldots,K\}$. By setting $k=1$ in Eq. (26), we further conclude that

[TABLE]

Now, fix $k\in\{2,\ldots,K\}$. We have, from Eq. (20), that

[TABLE]

where the last equality follows from the fact that

[TABLE]

Combining Eqs. (28) and (29), we have shown that the expressions in Eq. (20) are the unique invariant state of the fluid model, when $\alpha_{0}\in(0,\lambda_{1}/K]$.

We turn next to the second case of Theorem 5.3, and assume that $\alpha_{0}>\lambda_{1}/K$. First, suppose that $\mathcal{K}^{+}\neq\emptyset$. By the arguments leading to Eq. (27), which do not depend on the value of $\alpha_{0}$, we know that $\mathcal{K}^{+}=\{1\}$, and

[TABLE]

where both inequalities follow from the assumption that $\alpha_{0}>\lambda_{1}/K$. This contradicts with the assumption that $1\in\mathcal{K}^{+}$ and hence $q^{I}_{1}>\alpha_{0}$, and we conclude that $\mathcal{K}^{+}$ must be empty. This implies that $q^{I}_{k}\leq\alpha_{0}$ for all $k\in\mathcal{K}$, and by Eq. (20), we have that

[TABLE]

This completes the proof of Theorem 5.3. $\square$

The following theorem is the main result of this section, which would imply the first claim of Theorem 2.2.

Theorem 5.4

Fix $m\in\mathbb{N}$. The process $\overline{Q}^{m}(\cdot)$ admits a unique steady-state distribution, denoted by $\pi^{m}$. Suppose that $\beta_{m}\gg 1/m$ as $m\to\infty$. We have that

[TABLE]

where $q^{I}$ is the unique invariant state of the fluid model, and $\delta_{q^{I}}$ is the probability measure with unit mass on $q^{I}$. Furthermore, we have that

[TABLE]

The remainder of Section 5 is devoted to showing Theorem 5.4 and consists of three main parts, as illustrated in Figure 6.

Theorem 5.5 of Section 5.2 shows that the scaled recallable-reward process converges uniformly to a fluid solution over any finite time horizon, as $m\to\infty$. This is the more technical portion of the proof, and the key idea is to use a quasi-fluid process, one where the evolution of the reward process is deterministic while the choice process remains random, as an intermediary to bridge the gap between the fluid solution and the original stochastic process. 2. 2.

Theorem 5.8 of Section 5.3 shows that starting from any bounded initial condition, $q^{0}$, the fluid solution $q(q^{0},t)$ converges exponentially fast to the unique invariant state, as $t\to\infty$. The proof is centered around the evolution of a simple piece-wise linear potential function, which ensures that $q(q^{0},t)$ rapidly enters a suitable subset of the state space, from which point exponential convergence takes place. 3. 3.

Finally, in Section 5.4, by combining Theorems 5.5 and 5.8, we show that the sequence of steady-state scaled recallable rewards converges in $L_{1}$ to a unique limiting point concentrated on the invariant state of the fluid solution, completing the proof of Theorem 5.4.

5.2 Convergence to the Fluid Solution over Finite Horizon

In this subsection, we show that, with high probability, the scaled recallable reward process converges to the fluid solution uniformly over any finite horizon, as $m\to\infty$, as summarized in the following theorem (See plots $(a)$ through $(b)$ of Figure 5 for a sample-path example of how this convergence takes place.)

Theorem 5.5

Suppose that $\beta_{m}\gg 1/m$, as $m\to\infty$. Fix $q^{0}\in\mathbb{R}_{+}^{\mathcal{K}}$, and suppose that

[TABLE]

Let $q(q^{0},\cdot)$ be the unique fluid solution with initial condition $q^{0}$, as defined in Lemma 5.2. We have that, for all $T>0$,

[TABLE]

A main challenge in proving Theorem 5.5 is that the processes $Q^{m}(\cdot)$ and $C^{m}(\cdot)$ interact in an intricate way: the recallable reward vector influences how the subsequent choices are chosen, while the current choice in turn dictates the arrival rate of new rewards associated with each action. The main idea of the proof is to “‘disentangle” their interactions by means of an intermediate, quasi-fluid process, where the dynamics of the arrivals and departures of rewards becomes deterministic and “fluid-like”, while the choice process remains random.

We now construct the quasi-fluid process. Fix $q\in\mathbb{R}_{+}^{\mathcal{K}}$ and $c\in\mathcal{K}$, and define

[TABLE]

In words, $G_{k}(q,c)$ represents the rate of change in recallable rewards associated with action $k$ when the current recallable reward vector and action are $q$ and $c$, respectively. Define the scaled action process:

[TABLE]

Note that, unlike $\overline{Q}^{m}(\cdot)$, the scaling of $\overline{C}^{m}(\cdot)$ only occurs in time and not in space. We then construct a quasi-fluid solution, $\{V^{m}(t)\}_{t\in\mathbb{R}_{+}}$, by letting

[TABLE]

In words, $V^{m}(\cdot)$ corresponds to a system in which recallable rewards are governed by deterministic arrival and departure processes, as specified by Eq. (37), and where the choice process is scaled in time by a factor of $m$, but remains stochastic.

We note some basic properties of $V^{m}_{k}(\cdot)$ that will become useful. First, since $C^{m}(\cdot)$ is a Markov jump process and the function $G_{k}(\cdot,\cdot)$ uniformly Lipschitz continuous, analogously to Lemma 5.2, it is not difficult to show that $V^{m}(\cdot)$, expressed as a solution to the above integral equation, is uniquely defined almost surely. Also, it is not difficult to verify that

[TABLE]

where the right-hand side of the second inequality corresponds to the scenario where $\overline{C}^{m}(s)=k$ for all $s\in[0,t]$ and the negative drift term $(-q_{k})$ is absent from the expression of $G_{k}(q,c)$.

The proof of Theorem 5.5 proceeds in two main steps. We first show that $\{\overline{Q}^{m}(t)\}_{t\geq 0}$ converges to the quasi-fluid process, $\{V(t)\}_{t\geq 0}$, as $m\to\infty$, as is summarized in the following proposition.

Proposition 5.6

Fix $q^{0}\in\mathbb{R}_{+}^{\mathcal{K}}$, and suppose that

[TABLE]

Then,

[TABLE]

We then show that, when $\beta_{m}\gg 1/m$, the process $\{V^{m}(t)\}_{t\geq 0}$ converges to the fluid solution, $\{q(q^{0},t)\}_{t\geq 0}$:

Proposition 5.7

Suppose, in addition to the condition of Eq. (41) in Proposition 5.6, that $m\beta_{m}\to\infty$, as $m\to\infty$. For all $\epsilon>0$,

[TABLE]

Note, in particular, that Proposition 5.6 holds independently of the scaling of $\beta_{m}$, while Proposition 5.7 requires that the system be in the memory-abundant regime. Together, Propositions 5.6 and 5.7 imply Theorem 5.5.

The proof for Propositions 5.6 and 5.7 are given in Appendix C.1 and C.2, respectively. The main rationale in executing the proof of Theorem 5.5 via two steps is as follows. In Proposition 5.6, by fixing an arbitrary sample path of the choice process, $C(\cdot)$, the arrivals and departures from $Q^{m}(\cdot)$ become locally Poisson with a rate modulated by the current value of $C(t)$, and thus allowing us to establish a martingale property that will lead to the concentration of $Q^{m}(\cdot)$ around the quasi-fluid solution (Eq. (39)). With Proposition 5.6, the only remaining randomness in the quasi-fluid solution is in the choice process $C(\cdot)$ alone, and in Proposition 5.7 we show that its limiting behavior can be captured by a purely deterministic fluid solution in the memory-abundant regime. This two-step procedure is to be contrasted with analyzing the discrepancy between $Q^{m}(t)$ and the fluid solution directly, a more often adopted approach in the literature: in our model, the non-Markovian nature of $Q^{m}(t)$ implies that such differences do not immediately form a martingale.

Remark. We believe that there could be an alternative approach to establishing Theorem 5.5 by analyzing the limiting behavior of a discrete-time embedded Markov chain of the reward process $Q(\cdot)$ using stochastic approximation theory, and yet there are non-trivial difficulties along this direction. In particular, let $S_{i}$ be the $i$th update point (when the choice process $C(\cdot)$ can be updated), then the embedded reward process $\{Q[i]\stackrel{{\scriptstyle\triangle}}{{=}}Q(S_{i})\}_{i\in\mathbb{N}}$ can be shown to be a Markov chain, even though the original continuous-time process $Q(t)$ is not Markovian. One may then hope to invoke stochastic approximation theory to demonstrate that the scaled version of $Q[\cdot]$ converges to a fluid solution, such as the approach taken in Beggs, (2005). There are however several difficulties. First, the jumps in $Q[\cdot]$ can be unbounded due to the Poisson reward structure, while most standard results in the stochastic approximation literature require bounded updates between adjacent steps (e.g., Theorem 1 of Benveniste et al., (2012)); it is likely possible to remove this requirement but it will require a more delicate argument. Second, even assuming such convergence has been established, it would only apply to the update points themselves, and to obtain the type of uniform convergence over an entire continuous time interval in the form of Theorem 5.5 would require additional procedures to bound the maximal fluctuation of the reward process between event points. In light of these observations, we tend to believe that the approach adopted in this paper is more direct and easier to carry out.

5.3 Exponential Convergence of Fluid Solutions to Invariant State

In this section, we show that the fluid solution converges exponentially fast to the unique invariant state, starting from any bounded set of initial conditions.

Theorem 5.8

Fix $h_{0}>0$. There exist $a,b>0$, such that for all $u\in\mathbb{R}_{+}^{K}$, $\left\|u\right\|\leq h_{0}$,

[TABLE]

Main idea. The main difficulty in the proof of Theorem 5.8 is that, depending on the initial condition, the individual coordinates of $q(\cdot)$ may not converge to their equilibrium state monotonically, as can be seen in the trajectory of $q_{2}(\cdot)$ in the first three plots of Figure 5. The main idea behind our proof is to obtain a more accurate description of $q(\cdot)$ with the aid of the following piece-wise linear potential function:

[TABLE]

By analyzing the evolution of $g(q(\cdot))$, our proof shows that: (1) for every bounded initial condition of $q(\cdot)$, the fluid solution rapidly enters a desirable subset of the state space, and (2) starting from this subset exponential convergence to the invariant state takes place.

While potential functions are often used to establish convergence results, to the best of our knowledge, the use of the potential function $g(\cdot)$ in Eq. (44) is new. This is largely due to the fact that our potential function is specifically designed to mirror the total “weights” across all actions under the reward matching rule. In addition, our use of the potential function also diverges from the their conventional role in proving convergence results: we do not use $g(\cdot)$ directly for characterizing the rate of convergence; the primary role of $g(\cdot)$ is for controlling the position of the trajectory in the state-space. Once we are assured that the trajectory has entered a suitable subset of the state space, the rate of convergence can be more readily obtained via elementary arguments.

Proof of Theorem 5.8. Fix $u\in\mathbb{R}_{+}^{\mathcal{K}}$, such that $\left\|u\right\|\leq h_{0}$. For simplicity of notation, we will omit the dependence of the initial condition and write $q(\cdot)$ in place of $q(u,\cdot)$, whenever doing so does not cause any confusion. The proof of the theorem is divided into two parts, depending on the value of $\alpha_{0}$. For the first part of the proof, we will assume that $\alpha_{0}\leq\lambda_{1}/K$.

Let $g(\cdot)$ be the potential function defined in Eq. (44). We will first partition $\mathbb{R}_{+}^{\mathcal{K}}$ into three disjoint subsets, depending on the value of $g(\cdot)$. Let $\epsilon$

[TABLE]

and define

[TABLE]

We observe some useful properties of the fluid solutions.

Lemma 5.9

Fix $t>0$ so that all coordinates of $q(\cdot)$ are differentiable. The following holds:

If $q(t)\in\mathcal{Q}_{1}\cup\mathcal{Q}_{2}$, then

[TABLE]

Furthermore, there exists a constant $d_{1}>0$, such that if $q(t)\in\mathcal{Q}_{1}$, then

[TABLE] 2. 2.

Denote by $\mathcal{K}+(t)$ the coordinates over which $q(t)$ is at least $\alpha_{0}$:

[TABLE]

There exists a constant $d_{2}>0$, such that if $q(t)\in\mathcal{Q}_{2}\cup\mathcal{Q}_{3}$, then for all $k\in\mathcal{K}^{+}(t)$, $k\geq 2$,

[TABLE] 3. 3.

If $q(t)\in\mathcal{Q}_{3}$, then

[TABLE]

*Proof. *Suppose that $q(t)\in\mathcal{Q}_{1}\cup\mathcal{Q}_{2}$. We have that

[TABLE]

where step $(a)$ follows from the assumption that $q(t)\in\mathcal{Q}_{1}\cup\mathcal{Q}_{2}$. In a similar fashion, now suppose that $q(t)\in\mathcal{Q}_{1}$ and hence $g(q(t))\leq\lambda_{1}-\epsilon$. We obtain that

[TABLE]

This proves the first claim of the lemma.

For the second claim, fix $q(t)\in\mathcal{Q}_{2}\cup\mathcal{Q}_{3}$ and $k\in\mathcal{K}^{+}(t)$, $k\geq 2$. We have that

[TABLE]

where step $(a)$ follows from the assumption that $q(t)\in\mathcal{Q}_{2}\cup\mathcal{Q}_{3}$, and $(b)$ from the fact that $\lambda_{2}\geq\lambda_{k}$ for all $k\geq 2$, and $\epsilon\leq\frac{\lambda_{1}-\lambda_{2}}{2}$. This proves the second claim.

Finally, for the last claim, note that since $\alpha_{0}\leq\lambda_{1}/K$, the fact that $q(t)\in\mathcal{Q}_{3}$ implies that at least one component of $q(t)$ is no less than $\alpha_{0}$, and hence $\mathcal{K}^{+}(t)\neq\emptyset$. We have that

[TABLE]

where the strict inequality in step $(a)$ follows from the definition of $\mathcal{Q}_{3}$, the fact that $\mathcal{K}^{+}(t)$ is non-empty, and that $\lambda_{k}\leq\lambda_{1}$ for all $k$. This completes the proof of Lemma 5.9. $\square$

We proceed by considering two cases for the initial condition of $q(\cdot)$.

Case 1: $q(0)\in\mathcal{Q}_{1}\cup\mathcal{Q}_{2}$. Claim 3 in Lemma 5.9 implies that

[TABLE]

By Claim 1 of the same lemma, we hence conclude that $q_{1}(t)$ is non-decreasing in $t$ for all $t\in\mathbb{R}_{+}$. By the same claim, we observe that $\dot{q}_{1}(t)$ is bounded from below by the constant $d_{1}$ whenever $q(t)$ lies in $\mathcal{Q}_{1}$. Since $g(q(t))\geq q_{1}(t)$, we further conclude that in order for Eq. (56) to be valid, the amount of time $q(t)$ spends in $\mathcal{Q}_{1}$ must satisfy:

[TABLE]

which, combined with Eq. (56), implies that for all $t\geq 0$:

[TABLE]

Claim 2 of Lemma 5.9 states that if $q(t)\in\mathcal{Q}_{2}$, then any component of $q(t)$ in $\mathcal{K}^{+}(t)\backslash\{1\}$ will have a negative drift that is bounded away from zero. Fix $t\in\mathbb{R}_{+}$, and suppose there exists $k^{\prime}\in\mathcal{K}^{+}(t)\backslash\{1\}$. Since $\dot{q}_{k}(t)\leq\lambda_{1}$ for all $k$ and $t$, by Eq. (58), we have that

[TABLE]

Since the components of $q(t)$ must be non-negative, the above equation implies that for all $k\geq 2$,

[TABLE]

where $T_{0}=\frac{h_{0}+\lambda_{1}^{2}/d_{1}}{d_{2}}+\frac{\lambda_{1}}{d_{1}}.$

Eq. (60) will let us strengthen our previous observation. For all $t\geq T_{0}$, we have that

[TABLE]

Since $q_{1}(t)$ is non-decreasing in $t$ by the first claim of Lemma 5.9, we conclude that $g(q(t))$ is non-decreasing in $t$ after $T_{0}$. Combined with Eq. (57), this implies that

[TABLE]

where $T_{1}=T_{0}+\frac{\lambda_{1}}{d_{1}}$. Fix $t>T_{1}$. Observe that by the definition of $\epsilon$, we have that $q_{1}(t)\geq\alpha_{0}$ whenever $q(t)\in\mathcal{Q}_{2}$. We have that

[TABLE]

and hence

[TABLE]

where step $(a)$ follows from the fact that $q_{1}(t)\geq\alpha_{0}$. For step $(b)$, note that since $q(t)\in\mathcal{Q}_{2}$ and $g(q(t))=q_{1}(t)+(K-1)\alpha_{0}$, we have that

[TABLE]

The inequality in step $(b)$ thus follows from the fact that $1/(1-x)\geq 1+x$, for all $x\in[0,1)$, where we let $x=(q^{I}_{1}-q_{1}(t))/\lambda_{1}$.

Recall that, for $a>0$, the solution to the ordinary differential equation $\dot{x}(t)=-ax(t)$ is given by

[TABLE]

From Eq. (64), we conclude that for all $s>0$,

[TABLE]

where the last inequality follows from the fact that $q(T_{1})\in\mathcal{Q}_{2}$ and $q_{k}(T_{1})\leq\alpha_{0}$ for all $k\geq 2$, and hence $q_{1}(T_{1})\geq\lambda_{1}-(K-1)\alpha_{0}-\epsilon=q^{I}_{1}-\epsilon$. Let $a_{1}=q^{I}_{1}\vee\epsilon$, and $b_{1}=\alpha_{0}/\lambda_{1}$. Since $q_{1}(t)$ is non-decreasing by Lemma 5.9, from Eqs. (62) and (67), we have that if $q(0)\in\mathcal{Q}_{1}\cup\mathcal{Q}_{2}$, then

[TABLE]

We now show the convergence of $q_{k}(\cdot)$ for $k\geq 2$. Fix $k\in\{2,\ldots,K\}$. Define

[TABLE]

and recall from Theorem 5.3 that $q^{I}_{k}=\frac{\lambda_{k}}{\lambda_{1}}\alpha_{0}$, $k=2,\ldots,K$. By Eq. (60), we have that $q_{k}(t)\leq\alpha_{0}$ for all $t\geq T_{0}$. Therefore, for all $t\geq T_{1}$,

[TABLE]

Using the Taylor expansion of the function $f(x)=\frac{\alpha_{0}}{\lambda_{1}+x}$ around $x=0$ and noting that $\delta_{1}(t)\leq a_{1}\exp(-b_{1}t)$ (Eq. (68)), we have that there exists $T_{2}\geq T_{1}$, such that for all $t\geq T_{2}$,

[TABLE]

where $\gamma_{k}\stackrel{{\scriptstyle\triangle}}{{=}}\frac{2\lambda_{k}\alpha_{0}a_{1}}{\lambda_{1}}$, and we use the notation $x\in y\pm z$ to mean $y-z\leq x\leq y+z$. Given a fixed value of $\delta_{k}(T_{2})$, Eq. (71) implies that the value of $\delta_{k}(T_{2}+t)$ must lie between the solutions to the ODEs $\dot{x}(t)=-x(t)+\gamma_{k}\exp(-b_{1}t)$ and $\dot{x}(t)=-x(t)-\gamma_{k}\exp(-b_{1}t)$, with initial condition $x(0)=\delta_{k}(T_{2})$, respectively. It can be verified that the solution to the ODE $\dot{x}(t)=-x(t)+c_{1}\exp(c_{2}t)$, with initial condition $x(0)=x_{0}$, is given by $x(t)=\frac{c_{1}}{1-c_{2}}\exp(-c_{2}x)+\left(x_{0}-\frac{c_{1}}{1-c_{2}}\right)\exp(-x)$. Setting $c_{1}=\gamma_{k}$ and $c_{2}=b_{1}$, we thus conclude that there exist $\overline{a}_{k},{b}_{k}>0$, such that

[TABLE]

Note that $\dot{\delta}_{k}(t)$ is always bounded. We thus conclude hat there exists $a_{k}>0$ such that,

[TABLE]

Since the above equation holds for all $k\geq 2$, together with Eq. (68), we have proven Eq. (43) for the first case, by assuming that $q(0)\in\mathcal{Q}_{1}\cup\mathcal{Q}_{2}$.

Case 2: $q(0)\in\mathcal{Q}_{3}$. We now consider the second case where the initial condition, $q(0)$, belongs to the set $\mathcal{Q}_{3}$. Recall from Eq. (50) of Lemma 5.9 that when $q(t)\in\mathcal{Q}_{2}\cup\mathcal{Q}_{3}$, all components of $q(t)$ in $\mathcal{K}^{+}(t)$ exhibit a negative drift of magnitude at least $d_{2}$. Letting $T_{3}=h_{0}/d_{2}$, we thus conclude that one of the following scenarios must occur:

$q(s^{\prime})\in\mathcal{Q}_{1}\cup\mathcal{Q}_{2}$ for some $s^{\prime}\in[0,T_{3}]$. 2. 2.

$q(t)\in\mathcal{Q}_{3}$ for all $t\in[0,T_{3}]$, and there exists $s^{\prime}\in[0,T_{3}]$ such that

[TABLE]

For the first scenario, it is equivalently to “re-initializing” the system at time $t=s^{\prime}$ at a point in $\mathcal{Q}_{1}$. The proof for Case 1 presented earlier thus applies and Eq. (43) follows. In what follows, we will focus on the second scenario. Without loss of generality, it suffices to show the validity of Eq. (43) by fixing an initial condition:

[TABLE]

Not that, by the definition of $\mathcal{Q}_{3}$, this implies that

[TABLE]

We next show that $q_{1}(t)>q^{I}_{1}$ for all $t\in\mathbb{R}_{+}$. Fix $t>0$ such that

[TABLE]

Such $t$ exists because the set $\mathcal{Q}_{3}$ is open and $q(\cdot)$ is continuous. From Lemma 5.9, we know that if $q(t)\in\mathcal{Q}_{3}$, then any component in $\mathcal{K}^{+}(t)\backslash\{1\}$ must have a negative drift. It thus follows that

[TABLE]

We have that, whenever $q^{I}_{1}<q_{1}(t)<q^{I}_{1}+1$,

[TABLE]

where step $(a)$ follows from the fact that $q_{k}(t)\leq\alpha_{0}$ for all $k\geq 2$, and $(b)$ from $1/(1+x)\leq 1-x/2$ for all $x\in[0,1]$. In light of Eq. (66), the above inequality implies if the value of $q_{1}(t)-q^{I}_{1}$ is ever to be below $1$, then from that point on it will be bounded from below by an exponential function that is strictly positive. This shows that $q_{1}(t)>q^{I}_{1}$ for all $t>0$.

We now prove the exponential rate of convergence of $q_{1}(t)$ to $q^{I}_{1}$. Using again Eq. (79), we obtain

[TABLE]

where step $(a)$ follows from the fact that $q_{1}(t)>q^{I}_{1}$, and $(b)$ from $1/(1+x)\geq 1-x$ for all $x\geq 0$. Using the reasoning identical to that in Eqs. (64) through (68), we conclude that there exists $a_{1},b_{1}>0$, such that $|q_{1}(t)-q^{I}_{1}|\leq a_{1}\exp(-b_{1}t)$, for all $t\in\mathbb{R}_{+}.$

Finally, the fact that $q_{1}(t)>q^{I}_{1}$ implies that $g(q(t))>\lambda_{1}$, and hence $q(t)\in\mathcal{Q}_{3}$ for all $t>0$. By Eq. (78), this further implies that for any $k\geq 2$, $q_{k}(t)\leq\alpha_{0}$ for all $t\in\mathbb{R}_{+}$. The exponential rate of convergence of $q_{k}(t)$ for $k\in\{2,\ldots,K\}$ thus follows from the same argument as in Eqs. (70) through 73. This completes the proof of Theorem 5.8, assuming that $\alpha_{0}<\lambda_{1}/K$.

We now turn to the case where $\alpha_{0}>\lambda_{1}/K$. We will use a similar proof strategy by partitioning the state space based on the value of $g(q(t))$, albeit with a different partitioning. In particular, define the sets

[TABLE]

Note that by definition $g(q)\geq K\alpha_{0}$ for all $q\in\mathbb{R}_{+}^{\mathcal{K}}$, so the above sets constitute a partition of $\mathbb{R}_{+}^{\mathcal{K}}$. We have the following lemma.

Lemma 5.10

Fix $t>0$ so that all coordinates of $q(\cdot)$ are differentialble. If $q(t)\in\tilde{\mathcal{Q}}_{2}$, then there exists a constant $d_{3}>0$, such that $\frac{d}{dt}g(q(t))<-d_{3}$.

*Proof. *Suppose that $q(t)\in\tilde{\mathcal{Q}}_{2}$. Since $g(q(t))>K\alpha_{0}$, we have that $\mathcal{K}^{+}(t)\neq\emptyset$. We have that

[TABLE]

where step $(a)$ follows from the fact that $g(q(t))>K\alpha_{0}$, and $(b)$ from $\mathcal{K}^{+}(t)\neq\emptyset$. Finally, note that since $\alpha_{0}>\lambda_{1}/K$, $d_{3}$ is strictly positive. This completes the proof of the lemma. $\square$

Lemma 5.10 implies that if $q(0)\in\tilde{\mathcal{Q}}_{2}$, then we must have that

[TABLE]

Since the rate of change in every coordinate of $q(t)$ is bounded, in light of the above equation, it suffices to establish Eq. (43) by considering only the scenario where $q(t)\in\tilde{\mathcal{Q}}_{1}$ for all $t\in\mathbb{R}_{+}$. Fixing $k\in\mathcal{K}$, we have that

[TABLE]

where step $(a)$ follows from the fact that $q_{k}(t)\leq\alpha_{0}$ whenever $q(t)\in\tilde{\mathcal{Q}}_{1}$, $(b)$ from the fact that $g(q(t))=K\alpha_{0}$, and $(c)$ from the definition $q^{I}_{k}=\lambda_{k}/K$. In light of Eq. (66), Eq. (84) implies that

[TABLE]

Since the above equation holds for all $k$, we have thus established an exponential rate of convergence of $q(t)$ to $q^{I}$, as $t\to\infty$. This completes the proof of Theorem 5.8. $\square$

5.4 Convergence of Steady-State Distributions

We complete the proof of Theorem 5.4 in this subsection. We will begin by establishing a useful distributional upper bound on $Q^{m}_{k}(\infty)$. Fix $t\in\mathbb{R}_{+}$, and denote by $U^{m}_{\lambda}(t)$ the number of jobs in system at time $t$ in an $M/M/\infty$ queue with arrival rate $\lambda$ and departure rate $1/m$, and by $U^{m}_{\lambda}$ its steady-state distribution. It is well known that $U^{m}_{\lambda}$ is a Poisson random variable with mean $m\lambda$, and it follows that, for all $\lambda>0$,

[TABLE]

Define the set

[TABLE]

and the process

[TABLE]

We have the following lemma, whose proof is given in Appendix D.2.

Lemma 5.11

The process $\{\overline{W}^{m}(t)\}_{t\in\mathbb{R}_{+}}$ is positive recurrent, and $\overline{Q}^{m}_{k}(\infty)\preceq m^{-1}U^{m}_{\lambda_{1}}$, for all $k\in\mathcal{K}$.

Denote by $\pi_{W}^{m}$ the probability distribution over $\mathcal{W}$ (Eq. (87)) that corresponds to the steady-state distribution of $\overline{W}^{m}(\cdot)$. Lemma 5.11 can be used to to show that the sequence $\{\pi^{m}_{W}\}_{m\in\mathbb{N}}$ is tight, as formalized in the following result, whose proof is given in Appendix D.3.

Lemma 5.12

For every $x\in\mathbb{R}_{+}$, define $\overline{\mathcal{Q}}_{x}=\{q\in\mathbb{R}_{+}^{\mathcal{K}}:\max_{k\in\mathcal{K}}q_{k}\leq x\}\times\mathcal{K}$, $x\in\mathbb{R}_{+}.$ Then, for every $\epsilon>0$, there exists $M\in\mathbb{N}$ such that

[TABLE]

We say that $y$ is a limit point of $\{x_{i}\}_{i\in\mathbb{N}}$ if there exists a sub-sequence of $\{x_{i}\}$ that converges to $y$. It is not difficult to verify that the space $\mathcal{W}$ is separable as a result of both $\mathbb{R}_{+}^{\mathcal{K}}$ and $\mathcal{K}$ being separable. By Prohorov’s theorem, the tightness of $\{\pi_{W}^{m}\}_{m\in\mathbb{N}}$ thus implies that any sub-sequence of $\{\pi_{W}^{m}\}_{m\in\mathbb{N}}$ admits a limit point with respect to the topology of weak convergence. Let $\{\pi_{W}^{m_{i}}\}_{i\in\mathbb{N}}$ be a sub-sequence of $\{\pi_{W}^{m}\}_{m\in\mathbb{N}}$, and $\pi_{W}$ a limit point of the sub-sequence. Denote by $\pi$ and $\pi^{m_{i}}$ the marginals of ${\pi}_{W}$ and $\pi^{m_{i}}_{W}$ over the first $K$ coordinates (corresponding to $\overline{Q}^{m}(\cdot)$), respectively. In the remainder of the proof, we will show that $\pi$ necessarily concentrates on $q^{I}$.

We first show that ${\pi}$ is a stationary measure with respect to the deterministic fluid solution. To this end, we will use the method of continuous test functions (cf. Section 4 of Ethier and Kurtz, (2005)). Let $\overline{\mathcal{C}}$ be the space of all bounded continuous functions from $\mathbb{R}_{+}^{\mathcal{K}}$ to $\mathbb{R}_{+}$. We will demonstrate that, for all $f\in\overline{\mathcal{C}}$,

[TABLE]

where, from here onward, we will use the subscript, $\mathbb{E}_{\pi}(\cdot)$, to indicate the distribution of $q^{0}$.

Define $\mathcal{Q}^{m}=\{{x}/{m},\,x\in\mathbb{Z}_{+}^{\mathcal{K}}\}$, $m\in\mathbb{N}.$ Fix $f\in\overline{\mathcal{C}}$. Let $\overline{W}^{m}(0)$ be distributed according to $\pi^{m}_{W}$, and define

[TABLE]

Fix $t>0$. We have that

[TABLE]

Step $(a)$ follows from the triangle inequality, and $(b)$ from the fact that $Q^{m}(\cdot)$ is stationary when initialized according to $\pi^{m}$, and therefore,

[TABLE]

For step $(c)$, we note that since $q(q^{0},t)$ depends continuously on $q^{0}$ (Lemma 5.2), the function $g(x)=f(q(x,t)),\,x\in\mathbb{R}_{+}^{\mathcal{K}}$, belongs to $\overline{\mathcal{C}}$. Therefore, the step follows from the fact that $\pi^{m_{i}}$ converges weakly to $\pi$ as $i\to\infty$.

Fix $\epsilon>0$. There exists $M>0$ such that the the right-hand side of Eq. (93) satisfies

[TABLE]

where $(a)$ follows from the tightness property of Eq. (89), and step $(b)$ involves an argument that allows for interchanging the order of taking the limit and integration. We isolate step $(b)$ in the following lemma; the proof leverages Theorem 5.5 and is given in Appendix D.4.

Lemma 5.13

Fix a compact set $S\subset\mathbb{R}_{+}^{\mathcal{K}}$ and $f\in\overline{\mathcal{C}}$, we have that

[TABLE]

Fix $\delta>0$. We have that

[TABLE]

where $w_{f}$ is the modulus of continuity of $f$ in $S$: $w_{f}(S,\delta)=\sup_{x,y\in S,\|x-y\|\leq\delta}|f(x)-f(y)|$. Step $(a)$ follows from the fact that $\pi^{m}(\mathcal{Q}^{m})=1$, and $(b)$ from Eq. (226). Because $S$ is compact, $f$ is uniformly continuous in $S$ and hence $\lim_{\delta\to 0}w_{f}(S,\delta)=0$. The lemma follows by taking the limit as $\delta\to 0$ in Eq. (100). $\square$

Since Eq. (98) holds for all $\epsilon>0$, combining it with Eq. (93), we conclude that

[TABLE]

and this proves Eq. (90). We now show that Eq. (90) implies that $\pi=\delta_{q^{I}}$, i.e., $\delta_{q^{I}}$ is the unique invariant measure with respect to the dynamics induced by the fluid solution. Define the truncated norm:

[TABLE]

and the set

[TABLE]

Let $q^{0}$ be a random vector in $\mathbb{R}_{+}^{\mathcal{K}}$ distributed according to $\pi$. Suppose, for the sake of contradiction, that $\mathbb{E}(\|q^{0}-q^{I}\|_{M})\stackrel{{\scriptstyle\triangle}}{{=}}\mu_{0}>0$ for some $M>0$, where the expectation is taken with respect to the randomness in $q^{0}$. Fix $L>0$ such that $\mathbb{P}(q^{0}\notin\mathcal{R}_{L})\leq\frac{\mu_{0}}{4}\cdot\frac{1}{M}$. By Theorem 5.8, there exists $T>0$ such that $\sup_{x\in\mathcal{R}_{L}}\|q(x,T)-q^{I}\|_{M}<\mu_{0}/4$. Therefore,

[TABLE]

where the first inequality uses the fact that $\|q\|_{M}\leq M$ for all $q\in\mathbb{R}_{+}^{\mathcal{K}}$. Because $\mathbb{E}(\|q^{0}-q^{I}\|_{M})$ was assumed to be strictly positive, this is in contradiction with the stationarity of $\pi$ with respect to the fluid solution, which would imply that $\mathbb{E}(\|q(q^{0},T)-q^{I}\|_{M})=\mathbb{E}(\|q^{0}-q^{I}\|_{M})$ for all $T\in\mathbb{R}_{+}$. We thus conclude that

[TABLE]

which in turn implies that $\pi=\delta_{q^{I}}$. This proves Eq. (33).

Finally, to show Eq. (34), note that by Lemma 5.11, for all $m\in\mathbb{N}$ and $k\in\mathcal{K}$, the random variable $\overline{Q}^{m}_{k}(\infty)$ is non-negative and stochastically dominated by $m^{-1}U^{m}_{\lambda_{1}}$. Using similar steps as those in Eq. (225), it is not difficult to show that $\sup_{m\in\mathbb{N}}\mathbb{E}(|m^{-1}U^{m}_{\lambda_{1}}|^{2})<\infty$, which, implies the uniform integrability of $\{\overline{Q}^{m}_{k}\}_{m\in\mathbb{N}}$. In combination with Eq. (33), this proves the convergence in $L^{1}$ stated in Eq. (34). This completes the proof of Theorem 5.4 as well as the first claim of Theorem 2.2.

6 The Memory-Deficient Regime

We now prove the second claim of Theorem 2.2 concerning the memory-deficient regime, where $\beta_{m}\ll 1/m$, as $m\to\infty$. The statement is repeated below for easy reference.

Theorem 6.1

Suppose that $\beta_{m}\ll 1/m$, as $m\to\infty$. Then,

[TABLE]

where $q_{k}=\lim_{m\to\infty}\mathbb{E}\left(m^{-1}Q^{m}_{k}(\infty)\right)$.

Fix $m\in\mathbb{N}$. Let $S^{m}_{n}$ be the $n$th update point, and set $S^{m}_{0}\stackrel{{\scriptstyle\triangle}}{{=}}0$. Denote by

[TABLE]

the embedded discrete-time process associated with the choice process $\{C^{m}(t)\}_{t\in\mathbb{R}_{+}}$. The discrete-time processes $\{Q^{m}[n]\}_{n\in\mathbb{Z}_{+}}$ and $\{W^{m}[n]\}_{n\in\mathbb{Z}_{+}}$ are defined analogously.

The key to proving Theorem 6.1 hinges upon the following property: the discrete-time choice process, $\{C^{m}[n]\}_{n\in\mathbb{N}}$, becomes asymptotically Markovian in the limit as $m$ tends to infinity. Note that $\{C^{m}[n]\}_{n\in\mathbb{N}}$ is not Markovian for a finite $m$, because the choices are sampled from a distribution that is a function of the current recallable reward vector, which in turn depends on past choices. However, the dependence of the reward vector on past choices is greatly weakened in the memory-deficient regime: because the life span of a unit reward is so small compared to the time between two adjacent choices, by the time a new choice is to be made, the agent will have essentially forgotten all of the past rewards associated with any action other than the one she currently uses. Therefore, when $m$ is large, knowing the choice $C^{m}[n]$ for some $n\in\mathbb{N}$, we can predict with high accuracy the recallable reward vector at the $n+1$ update point, rendering the choice process approximately Markovian. The approximate Markovian property will allow us to explicitly characterize the steady-state distribution of $C^{m}[\cdot]$ in the limit as $m\to\infty$, which we then leverage to analyze the steady-state distribution of the recallable reward process, $\{Q^{m}(t)\}_{t\in\mathbb{N}}$. Following this line of thinking, our proof will consist of three main parts:

(Proposition 6.3) Fix $n\in\mathbb{N}$. We first show that, in the limit as $m\to\infty$, the value of the scaled recallable reward process at the $(n+1)$-th update point converges in probability to a deterministic vector that only depends on $C^{m}[n]$. 2. 2.

(Proposition 6.4) Using the convergence of the recallable reward vector, we derive an explicit expression for the steady-state distribution of the discrete choice process, $\{C^{m}[n]\}_{n\in\mathbb{N}}$, in the limit as $m\to\infty$. 3. 3.

Using the stationarity of the original recallable reward process, we combine the above two results to arrive at a characterization of the steady-state distribution of $\{Q^{m}(t)\}_{t\in\mathbb{R}_{+}}$, thus completing the proof of Theorem 6.1.

We begin by stating some basic properties of $Q^{m}[\cdot]$ and $C^{m}[\cdot]$, which will allow us to relate the steady-state behavior of these processes to their continuous-time counterparts; the proof of the following lemma is given in Appendix D.5.

Lemma 6.2

Fix $m\in\mathbb{N}$. The discrete-time process $\{W^{m}[n]\}_{n\in\mathbb{N}}$ is positive recurrent, whose steady-state distribution satisfies

[TABLE]

and

[TABLE]

For the remainder of the proof, we will fix $h:\mathbb{Z}_{+}\to\mathbb{Z}_{+}$ to be an increasing function, such that $1/\beta_{m}\gg h(m)\gg m$, and, in particular,

[TABLE]

Define the scaled discrete-time recallable reward process: $\overline{Q}^{m}[n]=m^{-1}Q^{m}(S^{m}_{n})=m^{-1}Q^{m}[n]$, $n\in\mathbb{N}$. The following proposition states that when $m$ is large, the value of $\overline{Q}^{m}[1]$ converges to a deterministic value that depends solely on $C^{m}[0]$: $\overline{Q}^{m}_{k}[1]$ is equal to $\lambda_{k}$ for $k=C^{m}[0]$, and zero, otherwise. In other words, the agent will have forgotten all past rewards other than those associated with her current choice. The proof is given in Appendix C.3.

Proposition 6.3

Define the $K\times K$ matrix $q^{*}$, where

[TABLE]

Suppose that the continuous-time process $\{W^{m}(t)\}_{t\in\mathbb{N}}$ is initialized at $t=0$ according to its steady-state distribution. Then, for all $i,k\in\mathcal{K}$,

[TABLE]

and

[TABLE]

Using Proposition 6.3, we will be able to obtain an explicit expression for the limit $\lim_{m\to\infty}\mathbb{P}(C^{m}[\infty]=k)$, as is stated in the following proposition.

Proposition 6.4

Define $p^{m}_{k}=\mathbb{P}(C^{m}[\infty]=k)$, $k\in\mathcal{K}.$ We have that

[TABLE]

where $z_{0}=\sum_{i\in\mathcal{K}}\left[\lambda_{i}\vee\alpha_{0}+(K-1)\alpha_{0}\right]$ is a normalizing constant.

*Proof. *Suppose that $\{W^{m}[n]\}_{n\in\mathbb{N}}$ is initialized at $n=0$ in its steady-state distribution. We have that, for all $i\in\mathcal{K}$,

[TABLE]

where $\mathcal{Q}^{m}\stackrel{{\scriptstyle\triangle}}{{=}}\frac{1}{m}\mathbb{Z}_{+}^{\mathcal{K}}$. By Eq. (112) in Proposition 6.3, Eq. (115) implies that, there exists a sequence of $K\times 1$ vectors $\{\delta^{m}\}_{m\in\mathbb{N}}$ with $\lim_{m\to\infty}\max_{i\in\mathcal{K}}|\delta^{m}_{i}|=0$, such that for all $i\in\mathcal{K}$ and $m\in\mathbb{N}$,

[TABLE]

where $q^{*}_{k,i}=\mathbb{I}(k=i)\lambda_{k}$. Note that by the stationarity of $C^{m}[\cdot]$, we have that $C^{m}[0]\stackrel{{\scriptstyle d}}{{=}}C^{m}[1]\stackrel{{\scriptstyle d}}{{=}}C^{m}[\infty]$. Therefore, by treating $p^{m}=(p^{m}_{1},p^{m}_{2},\ldots,p^{m}_{K})^{T}$ as a column vector, and defining the $K\times K$ matrix, $R$, where

[TABLE]

Eq. (116) can be written more compactly as

[TABLE]

It is not difficult to verify that $R$ is row-stochastic and corresponds to the transition kernel of a discrete-time, irreducible Markov chain over $\mathcal{K}$. Therefore, the equation $x=Rx$ admits a unique solution, $x^{*}$, in the $K$-dimensional simplex. Furthermore, it is not difficult to check that the following choice of $x^{*}$ satisfies $x^{*}=Rx^{*}$, and is hence the unique solution:

[TABLE]

Since $\lim_{m\to\infty}\max_{i\in\mathcal{K}}|\delta^{m}_{i}|=0$, combining Eqs. (118) and (119), we conclude that $p^{m}$ converges to $x^{*}$ coordinate-wise, as $m\to\infty$. This completes the proof of Proposition 6.4. $\square$

We are now ready to complete the proof of Theorem 6.1. Fix $m\in\mathbb{N}$. Suppose that we initiate the continuous-time process $\{W^{m}(t)\}_{t\in\mathbb{N}}$ in its steady-state distribution so that it is stationary. The stationarity implies that for all $k\in\mathcal{K}$,

[TABLE]

where $p^{m}_{i}$ is as defined in Proposition 6.4, and step $(a)$ follows from the fact that $C^{m}[\infty]=C^{m}(\infty)$ (Lemma 6.2). Taking the limit as $m\to\infty$, we have that

[TABLE]

where step $(a)$ follows from Eq. (113) in Proposition 6.3 and Eq. (114) in Proposition 6.4. This completes the proof of Theorem 6.1.

7 Proof of Theorem 2.1

We prove Theorem 2.1 in this section. Let $W^{m}[\cdot]=(C^{m}[\cdot],Q^{m}[\cdot])$ be the discrete-time embedded Markov chain defined in Eq. (107). By Lemma 6.2, $C^{m}[\infty]$ and $C^{m}(\infty)$ admit the same distribution, and it hence suffices to characterize the former. Because the limiting expressions for $C^{m}[\infty]$ in the memory deficient regime have already been established in Proposition 6.4, we will focus on the memory abundant regime, where $\beta_{m}\gg 1/m$. By Theorem 5.4, for all $\epsilon>0$

[TABLE]

where the first step follows from Eq. (109) in Lemma 6.2. Fix $k\in\mathcal{K}$. We have that

[TABLE]

where the last equality follows from Eq. (121) and the fact that $\frac{q_{k}\vee(m\alpha_{0})}{\sum_{i}q_{i}\vee(m\alpha_{0})}$ is uniformly bounded over $q\in\mathbb{R}_{+}^{\mathcal{K}}$. Eq. (122), along with the expression for $q^{I}$ in Theorem 5.3, leads to Eqs. (5) and (6), depending on the value of $k$ and $\alpha_{0}$. This completes the proof of Theorem 2.1.

8 Extensions and Generalizations

We examine in this section a number of extensions to the original model. We will focus on key ideas with the intention of illustrating possible future directions, and the discussion will be more exploratory in nature, and less formal than that of our main results.

8.1 Fluid Solutions with Polynomial Choice Models

We consider in this sub-section a generalization of the reward-matching choice rule. Recall that under reward matching, the probability of choosing action $k$ when the recallable reward vector is $Q$ is linearly proportional to $Q_{k}\vee\alpha$. A natural generalization would be to consider a polynomially weighted reward-matching rule, where the probability of choosing action $k$ is proportional to $\left(Q_{k}\vee\alpha\right)^{\eta}$, where $\eta>0$ is a fixed parameter.

An interesting question is how the agent’s behavior will depend on the value of $\eta$, and we will provide some preliminary analysis for this question in this sub-section. We will focus our attention on the invariant state(s) of the corresponding fluid solutions for the recallable rewards, as it serves as a good proxy for the stationary distribution of the pre-limit process in the memory-abundant regime, and from it the stationary choice distributions may also be inferred. The memory-deficient regime is in fact much easier to analyze, and we will postpone the discussion of that regime till the end of this sub-section (Section 8.1.1).

In the fluid limit, the polynomial reward matching rule would lead to a modified function, $p$ (originally defined in Eq. (14)), given by

[TABLE]

It turns out that the dynamics of the fluid model will highly depend on whether $\eta\leq 1$ or not, and we will divide our analysis into two cases as such.

Case 1: $\eta\leq 1$. The next proposition shows that when $\eta<1$ the fluid solutions admit a unique invariant state, and gives a characterization of its form.444Our original model already covers the case of $\eta=1$, and we shall therefore focus on the case where $\eta<1$. The proof of Proposition 8.1 is given in Appendix C.4.

Proposition 8.1

Fix $\alpha_{0}>0$ and $\eta\in(0,1)$. The fluid solutions admit a unique invariant state, $q^{I}$, whose expression depends on the value of $\alpha_{0}$, as follows.

Suppose that $\alpha_{0}>\lambda_{1}/K$. Then $q^{I}_{k}=\frac{\lambda_{k}}{K}$, for all $k\in\mathcal{K}.$ 2. 2.

Suppose that $\alpha_{0}\leq\lambda_{K}\frac{\lambda_{K}^{\frac{\eta}{1-\eta}}}{\sum_{i\in\mathcal{K}}\lambda_{i}^{\frac{\eta}{1-\eta}}}$. Then

[TABLE] 3. 3.

Suppose that $\lambda_{K}\frac{\lambda_{K}^{\frac{\eta}{1-\eta}}}{\sum_{i\in\mathcal{K}}\lambda_{i}^{\frac{\eta}{1-\eta}}}<\alpha_{0}\leq\lambda_{1}/K$. There exists $i^{*}\in\{1,\ldots,K-1\}$, such that $q^{I}_{i}\geq\alpha_{0}$ for $i=1,\ldots,i^{*}$, and $q^{I}_{i}<\alpha_{0}$ for $i=i^{*}+1,\ldots,K$. In particular, define the function

[TABLE]

Then, $i^{*}$ is the unique index that satisfies

[TABLE]

Furthermore, $q^{I}_{i^{*}}$ is the unique solution to the following equation in $\mathbb{R}_{+}$

[TABLE]

and the remaining coordinates of $q^{I}$ are given by:

[TABLE]

We can draw a few interesting observations from Proposition 8.1. The result suggests that the equilibrium behavior of the agent is largely consistent with the original model and our intuition: as $\eta$ decreases towards [math], the agent’s choice distribution tends to become uniform over the actions, since when $\eta$ is small, the agent’s choice probabilities depend weakly on the rewards of the actions. This is illustrated numerically in Figure 7-(a) via an example with two actions. The uniqueness also suggests that the stochastic approximation portion of our results (Theorem 2.2) should carry over to the regime of $\eta<1$ using the same arguments, without significant difficulty. Overall, these observations show that the insights from our original model should be fairly robust under small perturbation of $\eta$ in the direction towards [math].

Proposition 8.1 also reveals interesting new insights that are less obvious. Firstly, Item 1 of the proposition shows that the phenomenon of “complete oblivion” observed in the original model (see Theorem 2.2 and Section 2.1), where the agent chooses uniformly across actions as a result of a large exploration parameter $\alpha_{0}$, is not an artifact of $\eta$ being exactly equal to $1$, and in fact continues to exist whenever $\eta<1$. Moreover, the threshold on $\alpha_{0}$ that marks the beginning of such oblivion is the same as before, and does not depend on $\eta$.

Secondly, Item 2 of the proposition shows a regime not previously observed, and with a surprisingly clean expression: when $\alpha_{0}$ is very small, the probability of choosing action $k$ in the invariant state is exactly proportional to $\lambda_{k}^{\frac{\eta}{1-\eta}}$, and this expressions, remarkably, does not depend on $\alpha_{0}$ (to be contrasted with Eq. (5) which does depend on $\alpha_{0}$). This result would appear to lead to a contradiction with Theorem 2.2: by increasing $\eta$ towards $1$, Eq. (124) suggests that the agent will eventually concentrate $100$% of her efforts on the best action and the degree of such concentration is independent of $\alpha_{0}$. In other words, how is it possible that the agent may choose the optimal action more frequently when $\eta<1$ than when $\eta=1$? To resolve this puzzle, we note that the regime in Item 2 requires $\alpha_{0}\leq\lambda_{K}\frac{\lambda_{K}^{\frac{\eta}{1-\eta}}}{\sum_{i\in\mathcal{K}}\lambda_{i}^{\frac{\eta}{1-\eta}}}$, and the right-hand side, which is also equal to the invariant state of the worst action, $q^{I}_{K}$, tends to [math] as $\eta\to 1$. Therefore, as $\eta$ approach $1$ from below, the regime in Item 2 remains valid only if $\alpha_{0}$ vanishes accordingly, thus resolving the paradox.

There is another important consequence of Item 2. Suppose that $\alpha_{0}$ is fixed at a sufficiently small value, say, $\frac{1}{10}\cdot\frac{\lambda_{K}^{2}}{\sum_{i}\lambda_{i}}$. Then, Item 2 becomes the only valid regime for all sufficiently small $\eta$. As such, Eq. (124) provides a precise, quantitative description as to how the agent’s choice probabilities become uniform, as $\eta\to 0$, namely, that the probability of choosing action $k$ will be equal to $\frac{\lambda_{k}^{\frac{\eta}{1-\eta}}}{\sum_{i\in\mathcal{K}}\lambda_{i}^{\frac{\eta}{1-\eta}}}$ for all sufficiently small $\eta$.

Finally, Item 3 of the proposition addresses the remaining scenarios, where $\alpha_{0}$ is neither too small nor too big. Unfortunately, we no longer have a closed-form expression for this intermediate regime. Nevertheless, the invariant state can still be evaluated without much difficulty, and all coordinates of $q^{I}$ can be derived in closed-form as a function of $i^{*}$ and $q^{I}_{i^{*}}$.

Case 2: $\eta>1$. We now look at the second case, where $\eta>1$. The dynamics of the fluid solutions turn out to be more complex than when $\eta<1$, and we do not yet have a good understanding. To illustrate the complexities that one would encounter in this regime, we next show a simple example demonstrating that the fluid solutions could admit multiple invariant states.

Consider the a setting with two actions ($K=2$), where $\eta>1$ and $\alpha_{0}<\lambda_{1}^{\frac{1}{1-\eta}}/(\lambda_{1}^{\frac{\eta}{1-\eta}}+\lambda_{2}^{\frac{\eta}{1-\eta}})$. The drifts of the fluid solutions for this example are visualized in Figure 7-(b). The fluid solutions admit at least three distinct invariant states, marked by the red dots in Figure 7-(b). First, one can verify that the vector $q^{(1)}$, where $q^{(1)}_{1}=\frac{\lambda_{1}^{\frac{1}{1-\eta}}}{\lambda_{1}^{\frac{\eta}{1-\eta}}+\lambda_{2}^{\frac{\eta}{1-\eta}}}$ and $q^{(1)}_{2}=\frac{\lambda_{2}^{\frac{1}{1-\eta}}}{\lambda_{1}^{\frac{\eta}{1-\eta}}+\lambda_{2}^{\frac{\eta}{1-\eta}}},$ constitutes an invariant state (labeled $\left<1\right>$ in Figure 7-(b)). However, $q^{(1)}$ is not the unique invariant state, and nor is it stable: a perturbation along the direction of $(\epsilon,-\epsilon)$ or $(-\epsilon,\epsilon)$, where $\epsilon>0$ is a small constant, can induce the fluid solution to move towards one of the other two (stable) invariant states, $q^{(2)}$ (labeled $\left<2\right>$ in Figure 7-(b)) and $q^{(3)}$ (labeled $\left<3\right>$ in Figure 7-(b)), respectively. Here, $q^{(2)}$ satisfies $q^{(2)}_{1}>\alpha_{0}>q^{(2)}_{2}$, $\lambda_{1}\frac{(q^{(2)}_{1})^{\eta}}{(q^{(2)}_{1})^{\eta}+\alpha_{0}^{\eta}}=q^{(2)}_{1}$, and $q^{(2)}_{2}=\lambda_{2}\frac{\alpha_{0}^{\eta}}{(q^{(2)}_{1})^{\eta}+\alpha_{0}^{\eta}}$. Analogous to $q^{(2)}$, the state $q^{(3)}$ satisfies $q^{(3)}_{1}<\alpha_{0}<q^{(3)}_{2}$, $\lambda_{2}\frac{(q^{(3)}_{2})^{\eta}}{(q^{(3)}_{2})^{\eta}+\alpha_{0}^{\eta}}=q^{(3)}_{2}$ and $q^{(3)}_{1}=\lambda_{1}\frac{\alpha_{0}^{\eta}}{(q^{(3)}_{2})^{\eta}+\alpha_{0}^{\eta}}$.

We offer some speculation as to why the multiplicity of invariant states emerges when $\eta>1$. In this case, the agent leans disproportionally more towards actions with higher rewards, which means that sufficiently substantial advantages in the initial recallable rewards $q(0)$ could lead an action $i$ to permanently dominate another action $j$, even if $\lambda_{i}<\lambda_{j}$. This logic suggests that, in principle, if the reward rates do not vary significantly across actions, then any measure that concentrates on a single action could be a stable invariant state for the choice process, so long as the fluid solution starts in a state where the action has a substantial amount of initial rewards compared to other actions; this seems to explain the two stable invariant states in Figure 7-(b), $\left<2\right>$ and $\left<3\right>$. The unstable invariant state $\left<1\right>$, on the other hand, seems to stem from a different type of dynamics: it is a point where the action with the superior reward rate (in this case, $\lambda_{1}$) has fewer recallable rewards ($q_{1}(t)$), and the disadvantage in recallable rewards is exactly naturalized by the advantage in reward rate. The point is unstable because small perturbations could easily break this balance, inducing the advantage in the reward rate to dominate that of recallable rewards or vice versa.

What does this mean for the pre-limit, stochastic system? Note that while the fluid solutions can admit multiple invariant states when $\eta>1$, the steady-state distribution for any pre-limit stochastic system ($m<\infty$) is always unique: the exponential decay of recallable rewards in the pre-limit system ensures that $Q(\cdot)$ is positive recurrent, as it will return infinitely often to the state [math]. Nevertheless, the proceeding analysis of the fluid solutions suggests that the stochastic system will likely experience a qualitative change as well once $\eta$ is greater than $1$: when $m$ is large, instead of converging over time to the neighborhood of a single configuration, as is the case when $\eta\leq 1$, $Q(\cdot)$ is likely to alternate between multiple different configurations, each corresponding to one of the multiple invariant states in the fluid solutions.

8.1.1 General Choice Models

At a higher level, the reward matching rule is a special case of a broader family of choice heuristics, in which action $k$ is chosen with a probability proportional to $w(Q_{k}(t)\vee\alpha)$, where $w:\mathbb{R}_{+}\to\mathbb{R}_{+}$ is a non-decreasing weight function. The polynomial reward-matching rule studied earlier corresponds to $w(x)=x^{\eta}$, for some $\eta>0$. One may also consider $w(x)=\exp(cx)$ for some constant $c>0$, the sampling procedure becomes reminiscent of the celebrated logit model in choice theory (cf. Chapter 3, Train, (2009)) as well as the exponential-weight algorithm in the multi-arm bandit literature (cf. Auer et al., (2002)).

Can we extend our results beyond the polynomial reward-matching rule to include this even broader family of choice models? On the positive side, it is not difficult to check that the same analysis for the memory-deficient regime can be readily extended to these models: the limiting probability of choosing action $k$ would become proportional to $w(\lambda_{k})\vee\alpha_{0}+(K-1)\alpha_{0}$, i.e., with $\lambda_{k}$ in our original result being replaced by $w(\lambda_{k})$. Note that if we had the freedom of choosing $w(\cdot)$, then applying a highly skewed weight function, e.g., $w(x)=10^{20^{x}}$, would induce the agent to almost always choose the best action in steady state, even in the memory-deficient regime. However, the downside is that with a skewed weight function, the agent is highly unlikely to give up the current choice at an update point, regardless of its reward rate. It will thus take a long time for the agent’s time-average behavior to approach that of the steady state, resulting in bad transient performance. A similar trade-off seems to be present in the context of the exploration parameters, $\alpha_{0}$, where a smaller $\alpha_{0}$ tends to improve steady-state performance at the expense of a slower convergence to the steady-state dynamics. It would be interesting to obtain a more precious and rigorous understanding of the trade-off between the agent’s steady-state versus transient performance.

On the negative side, extensions to general choice models appear to be significantly more difficult in the memory-abundant regime, as is evidenced by the proceeding analysis of the polynomial reward-matching model. Making progress on this front is likely to require more sophisticated analysis to cope with the complex dynamics that will arise in fluid solutions, such as when $\eta>1$ in the polynomial reward-matching rule.

8.2 Non-Exponential Lifespan Distributions

We have thus far assumed that the lifespans of rewards follow an exponential distribution, and it would be interesting to see whether the qualitative insights from our model would continue to hold under other, non-exponential lifespan distributions. In this subsection, we experiment with two other lifespan distributions: in the first case, we set the lifespan to be a constant, $m$, and in the second case, the lifespan follows a (heavy-tailed) Pareto distribution with scale $m/2$, shape $2$, and mean $m$. Figure 8 shows the steady-state distribution of choices under these two scenarios, with $m=200$. Interestingly, comparing the results in Figure 8 to those from original model (Figure 4-(b)), we see that the distribution of choices is largely insensitive to the choice of lifespan distribution, and the theoretical predictions made in Theorem 2.2 appear to hold even under these non-exponential distributions.

8.3 Heterogeneous Memory Decay Rates

As was alluded to in Example 2 in the Introduction, we may also consider the case where the memory decay rates are not constant across actions, so that the rewards associated with action $k$ depart at rate $\mu_{k}$. This generalization is useful for modeling service system where the rate of customer departure or attrition depends on the service type a customer is associated with.

Analogous to the case of uniform decay rate, the scaling of Section 1.2 corresponds to setting $\mu=\mu^{0}_{k}m^{-1}$ in the $m$th system, where $\{\mu^{0}_{k}\}_{k\in\mathcal{K}}$ are positive constants. Under this scaling, it is not difficult to verify that both Theorems 2.1 and 2.2 extend to this more general case, if we replace every occurrence of $\lambda_{k}$ with a decay-rate-weighted version, $\lambda_{k}/{\mu^{0}_{k}}$. In other words, as far as the limiting system is concerned, the version of the problem with heterogeneous memory decay rates is equivalent to one with uniform decay rates but appropriately weighted reward rates. We should note that this equivalence only holds in the limit as $m\to 0$, and is not exact in a pre-limit system with a finite $m$.

9 Discussion

We have proposed in this paper a stochastic action-reward model for studying the impact of imperfect memory in dynamic decision making. In the limiting regimes where the agent’s memory span is large, our main results provide exact expressions for the steady-state choice probabilities of an agent following the so-called reward-matching rule, and demonstrate that these probabilities are highly sensitive to the relative scaling between two parameters: the rate of memory decay, $\mu$, and the rate at which the agent makes new choices, $\beta$.

There is a number of potentially interesting extensions of our formulation. For instance, the current paper focuses on the regime where the rate of memory decay $\mu\to 0$, and alternatively, one may also look at an opposite limiting regime where $\mu$ tends to infinity, i.e., the agent becomes extremely forgetful and her memory span approaches zero. Note that since the total amount of recallable rewards in system is on the order of $\mu^{-1}$ (regardless of the value of $\beta$), in this limit we should see very few recallable rewards at any point in time and the process of recallable rewards frequently hitting the zero state. It is therefore reasonable to assume that the reward-matching rule will perform poorly when $\mu$ is very large, with the agent choosing all actions essentially uniformly at random. Unfortunately, we do not yet have the tools needed to characterize this regime, since the fluid approximations developed in this paper heavily rely on mean-field-type evolution of a large amount of recallable rewards and are unlikely to be applicable when $\mu$ is large.

The present reward-matching model is symmetric, in the sense that the agent does not have any inherent preferences of one action over another. One way to relax this assumption is to allow the exploration parameter $\alpha$ to depend on the action, so that action $i$ will be sampled with probability $\propto(Q_{k}(t)\vee\alpha^{(k)})$. In the memory-deficient regime, it would not be difficult to show that this results in a steady-state choice distribution where action $k$ is chosen with probability proportional to $\lambda_{k}\vee\alpha_{0}^{(k)}+\sum_{i\neq k}\alpha_{0}^{(k)}$ . The memory-abundant regime will become more challenging to analyze, because the heterogeneity of $\alpha^{(k)}$ would in general break the monotonicity of the invariant-state recallable rewards, since now an action with a higher reward rate may not necessarily have a higher recallable reward in the invariant state. It would be an interesting future direction to better understand how to incorporate inherent preference asymmetries into the formulation.

While the present paper focuses largely on the theory, there appears to be several interesting potential applications of our model. First, the reward matching rule can be viewed as a simplified cognitive model to capture humans’ unconscious reinforcement behavior, and it will be interesting to see if the theory could be used to make predictions on a consumer’s long-run choice behavior when switching among similar products or services (eg., Example 1 of Section 1), or to investigate the effect of memory loss on a player’s learning and performance in repeated games similar to those studied by Erev and Roth, (1998). Second, memory loss can be interpreted in a more metaphorical sense, and be used to model the departure of customers (eg., Example 2 of Section 1). Cast in this light, the reward matching policy may act as a simple and intuitive heuristic in a manager’s algorithmic toolbox for dynamically choosing product offerings or demographic targets for advertisements. Finally, the dynamics of the recallable rewards in our system resemble that of a queueing network. For instance, the departures of rewards for each individual action function similarly to service completions in an infinite-server queue. There has been a growing literature in recent years on queuing systems where customers make their own choices regarding abandonments or the types of services they would like to receive (cf. Hassin and Haviv, (2003), Pender et al., (2016), Ding et al., (2016), Dong et al., (2018)), and it would be interesting to see whether our model can be applied to the study of (strategic) choice behavior in these service systems.

While memory loss is regarded as a given constraint in the current work, one could also ask whether it would be beneficial to artificially induce memory loss even when perfect recall is feasible. For instance, it has been observed in the multi-arm bandit literature that policies with built-in regularization that penalizes distant past experience can achieve better regret with time-varying, or adversarially chosen, parameters (cf. Garivier and Moulines, (2008), Van Erven and Kotl, (2014), Besbes et al., (2015), Keskin and Zeevi, (2016)). This is because these policies tend to be more adaptive to the environment and not “weighed down” by past experience. While our theorems do not directly apply to time-varying reward rates, Theorem 5.8 shows that the fluid solution converges exponentially quickly to the invariant state from any bounded initial condition, suggesting that the reward matching rule could be an attractive algorithm for dynamic learning in non-stationary environments.

Finally, at a higher level, there appear to be other interesting ramifications of “memory loss” in organizations to be explored. For instance, departures of employees could lead to the loss of skills and expertise (cf. Benkard, (2000)), and such organizational forgetting could potentially affect a firm’s efficiency in a significant way.

10 Acknowledgment

The authors would like to thank the anonymous referees at Mathematics of Operations Research for their comments and input, and Professors Steven Callander, Dana Foarta, J. Michael Harrison, David M. Kreps, Mihalis Markakis, Michael Ostrovsky, Daniel Russo, Daniela Saban and Takuo Sugaya for the insightful discussions and feedback on the manuscript. Yun would like to acknowledge the support from the National Research Foundation of Korea (NRF) grant No. 2019028324, funded by the government of Korea (MSIT).

Appendix A Comparison to Perfect Memory

We discuss in this appendix what could happen if there were no memory decay in our model, i.e., if $\mu=0$, and why it is different from the limiting regime considered in this paper, with $\mu\to 0$.

When $\mu=0$, all recallable rewards remain in the system indefinitely. If we view the recallable rewards sampled at the update points as a discrete-time process, and in addition set the exploration parameter $\alpha$ to 0, then our model becomes essentially the same as the choice process analyzed by Beggs, (2005), where it is shown that the probability of choosing the best action converges to one as $t\to\infty$ under Luce’s rule, as long as each action is associated with some strictly positive initial rewards. If $\alpha$ is a positive constant, because there is no memory decay, the effect of $\alpha$ disappears as soon as the rewards for all actions exceed $\alpha$, and the same conclusion should hold.

Therefore, one would expect that when $\mu=0$ and $\alpha\geq 0$, the choice probability under the reward-matching rule will concentrate on the best action as $t\to\infty$, regardless of the update rate, $\beta$. This is however different from the conclusion of Theorem 2.1, which shows the existence of two distinct limiting steady-state probabilities, one of which does not exhibit concentration on the best action. These observations thus suggest that our scaling regime do capture unique effects of imperfect memory. This is perhaps not too surprising in hindsight: if we had set $\mu$ to zero, any positive update rate $\beta$ would become, by definition, significantly greater than $\mu$, and hence the second (memory-deficient) regime in Theorems 2.1 and 2.2 could not have appeared when $\mu=0$.

Appendix B Technical Preliminaries

Proposition B.1** (Doob’s inequality)**

*(cf. Section 12.6, Grimmett and Stirzaker, (2001))

Let $\{X(t)\}_{t\geq 0}$ be a discrete- or continuous-time non-negative submartingale. Fix $T\geq 0$. We have that*

[TABLE]

Proposition B.2** (Gronwall’s lemma)**

(cf. Section 1.3, Ames and Pachpatte, (1997)) Let $f,b:\mathbb{R}_{+}\to\mathbb{R}_{+}$ be continuous and non-negative functions, and let $a:\mathbb{R}_{+}\to\mathbb{R}_{+}$ be a continuous, positive and non-decreasing function. If $f(t)\leq a(t)+\int_{0}^{t}b(s)f(s)ds$, for all $t\in[0,T],$ then

[TABLE]

Appendix C Proofs of Propositions

C.1 Proof of Proposition 5.6

The proof of Proposition 5.6 is largely based on maximal inequalities for continuous-time martingales, and the main steps follow the approach developed in Kurtz, (1978). Define $X^{m}(t)\in\mathbb{R}_{+}^{\mathcal{K}}$, where

[TABLE]

The proof proceeds in two stages: we first show, via Gronwall’s lemma (Proposition B.2 in Appendix B), that bounding the deviation of $\overline{Q}^{m}_{k}(\cdot)$ from $V^{m}_{k}(\cdot)$ can be reduced to bounding the magnitude of $X^{m}_{k}(\cdot)$. We then show that this can be accomplished by writing $X^{m}_{k}(\cdot)$ as a sum of two continuous-time martingales. From the definition of $\{V^{m}(t)\}_{t\geq 0}$, we have that

[TABLE]

For step $(a)$, we observe that for all $c,k\in\mathcal{K}$, $G_{k}(\cdot,c)$ is a $1$-Lipschitz continuous function. We now apply Gronwall’s lemma (Proposition B.2 in Appendix B) to Eq. (135), with $a(t)=\left\|\overline{Q}^{m}(0)-q^{0}\right\|+\left\|X^{m}(t)\right\|$, $b(t)=1$, and $f(t)=\left\|\overline{Q}^{m}(t)-V^{m}(t)\right\|$, and obtain that

[TABLE]

Because we have assumed that $\lim_{m\to\infty}\mathbb{P}\left(\left\|\overline{Q}^{m}(0)-q^{0}\right\|>\epsilon\right)=0$ for all $\epsilon>0$, to prove Lemma 5.6, it therefore suffices to show that

[TABLE]

We now prove Eq. (137) by expressing $X^{m}_{k}(\cdot)$ as the sum of two continuous-time martingales corresponding to the arrivals and departures of rewards, respectively. Fix $t\in\mathbb{R}_{+}$ and $k\in\mathcal{K}$, and denote by $A^{m}_{k}(t)$ and $D^{m}_{k}(t)$ the amount of rewards associated with action $k$ that have arrived and departed, respectively, during the interval $[0,t]$. In particular, we can write

[TABLE]

We have that

[TABLE]

For the remainder of the proof, we will focus on showing that, for all $\epsilon>0$,

[TABLE]

In light of Eq. (140), the above two equations imply the validity of Eq. (137), which proves Proposition 5.6. The proofs for both Eqs. (141) and (142) hinge upon the following technical result, which states that the sample path of a certain time-inhomogenous Poisson process stays close to its mean, with high probability. Note that a similar result for uniform-rate Poisson processes can be found in Lemma 7.6 of Massey and Whitt, (1998). The proof of the lemma is given in Appendix D.6, and uses a similar line of argument as that of Theorem 2.2 in Kurtz, (1978).

Lemma C.1

Fix $l\in\mathbb{Z}^{K+1}$ and $T\in\mathbb{R}_{+}$. Let $\{N(t)\}_{t\geq 0}$ be the counting process where $N(t)$ is the number of times in $[0,t]$ that the process $W^{m}(\cdot)$ jumps from state $w$ to $w+l$, for some $w\in\mathbb{Z}^{K+1}$. Denote by $\psi:\mathbb{Z}^{K+1}\to\mathbb{R}_{+}$ the corresponding rate function of $N(\cdot)$, so that the instantaneous transition rate of $N(\cdot)$ at time $t$ is equal to $\psi(W^{m}(t))$. For all $\epsilon,\phi>0$, we have that

[TABLE]

where $h(x)=(1+x)\log(1+x)-x$.

We now prove Eqs. (141) and (142). In the context of Lemma C.1, $A_{k}(\cdot)$ is the counting process with $l=e_{k}$, where $e_{k}$ is the vector $(K+1)\times 1$ vector whose $k$th coordinate is $1$ and all other coordinates zero, corresponding to an arrival to $Q^{m}_{k}(\cdot)$. The rate of $A_{k}(\cdot)$ at time $t$ is equal to $\lambda_{k}\mathbb{I}(C^{m}(t)=k)$, which is bounded from above by $\lambda_{k}$ for all $t\in\mathbb{R}_{+}$. By applying Lemma C.1, with $\psi(W^{m}(t))=\lambda_{k}\mathbb{I}(C^{m}(t)=k)$ and $\phi=\lambda_{k}$, we have that $\mathbb{P}\left(\sup_{0\leq t\leq T}\psi(W^{m}(t))>\phi\right)=0$, and for all $\epsilon>0$,

[TABLE]

The proof of Eq. (142) uses essentially the same idea, but the argument needs to be more delicate due to the fact that the rate of the counting process $D^{m}_{k}(t)$, which is equal to $Q^{m}_{k}(t)\mu=Q^{m}_{k}(t)m^{-1}$, is not bounded over the state space. Therefore, we first derive an upper bound on the tail probabilities of $Q^{m}_{k}(t)$, as follows. Let $\psi(\cdot)$ be the rate function for $D_{k}(\cdot)$ as in Lemma C.1, and $\phi=2q^{0}_{k}+(1+\epsilon)\lambda_{k}T$. Fixing $\epsilon>0$, we have that, for all $r\in\mathbb{Z}_{+}$,

[TABLE]

where step $(a)$ follows from the fact that $A^{m}_{k}(\cdot)$ is independent of $Q^{m}_{k}(0)$, and hence the maximum in the second term is attained by setting $r=2mq^{0}_{k}$. Step $(b)$ follows from $\lambda_{k}\mathbb{I}(C^{m}(s)=k)\leq\lambda_{k}$ for all $t\in\mathbb{R}_{+}$, and $(c)$ from the inequality in Eq. (143), by replacing $\epsilon$ with $\epsilon\lambda_{k}T$.

We are now ready to establish Eq. (142):

[TABLE]

where step $(a)$ follows from Lemma C.1, and $(b)$ from Eq. (146). Note that $\phi,\epsilon$ and $T$ are positive constants, and by our assumption, $\lim_{m\to\infty}\mathbb{P}(|m^{-1}Q^{m}_{k}(0)-q^{0}_{k}|>\delta)=0$ for all $\delta>0$. Therefore, Eq. (142) follows by taking the limit in the above inequality as $m\to\infty$. This completes the proof of Proposition 5.6.

C.2 Proof of Proposition 5.7

For the purpose of this proof, we will use an alternative representation of the fluid solutions using integral equations. Define the drift function, $F:\mathbb{R}_{+}^{\mathcal{K}}\to\mathbb{R}_{+}^{\mathcal{K}}$:

[TABLE]

where $p(\cdot)$ is defined in Eq. (14). It can be verified from the definition of $p(\cdot)$ that there exists $l_{F}\in\mathbb{R}_{+}$ such that $F_{k}(\cdot)$ is $l_{F}$-Lipschitz continuous for all $k\in\mathcal{K}$. Fix $q^{0}\in\mathbb{R}_{+}^{\mathcal{K}}$, let $\{q(t)\}_{t\in\mathbb{R}_{+}}$, $q(t)\in\mathbb{R}_{+}^{\mathcal{K}}$, be a solution to the following integral equation:

[TABLE]

Similar to Lemma 5.2, it is not difficult to show that the function $q(\cdot)$ defined in Eq. (149) exists, is unique, and coincides with the fluid solution with initial condition $q^{0}$. We have that

[TABLE]

where the last inequalities come from the fact that $F_{k}(\cdot)$ is $l_{F}$-Lipschitz continuous for all $k\in\mathcal{K}$. In a manner analogous to Eq. (135) from the proof of Proposition 5.6, by applying Gronwall’s lemma (Proposition B.2 in Appendix B), we have that, for all $\epsilon>0$,

[TABLE]

Therefore, in order to establish Proposition 5.7, it suffices to show that

[TABLE]

In what follows, we will show (154) by using the discrete-time embedded process of $C^{m}(\cdot)$ and analyzing the system dynamics at the times when new choices are chosen. We begin by introducing some notation. Fixing $i\in\mathbb{Z}_{+}$, we denote by $S_{i}$ the $i$th update point, i.e., time of the $i$th update in $C^{m}(\cdot)$, with $S_{0}\stackrel{{\scriptstyle\triangle}}{{=}}0$, and

[TABLE]

Note that $\{\tau_{i}\}_{i\in\mathbb{N}}$ are independent exponential random variables with mean $(m\beta_{m})^{-1}$. Let $\{\overline{Q}^{m}[i]\}_{i\in\mathbb{N}}$ be the discrete-time process, where $\overline{Q}^{m}[i]$ corresponds to the value of $\overline{Q}^{m}(\cdot)$ immediately following the $i$th update point:

[TABLE]

and $\{Z^{m}_{k}[i]\}_{i\in\mathbb{N}}$ be the process of indicator variables:

[TABLE]

That is, $Z^{m}_{k}[i]=1$ if action $k$ is selected on the $i$th update point. Finally, let $\{\widehat{Q}^{m}(t)\}_{t\geq 0}$ be a right-continuous piece-wise constant process which coincides with $\overline{Q}^{m}(\cdot)$ at the points $\{S_{i}/m\}_{i\in\mathbb{N}}$:

[TABLE]

Fix $k\in\mathcal{K}$, and denote by $I_{t}$ the number of updates in $C^{m}(\cdot)$ by time $t$, i.e.,

[TABLE]

By the triangle inequality, we have that

[TABLE]

We now derive upper bounds on tail probabilities for each term on the right-hand side of Eq. (161). For the first term, define

[TABLE]

Fix $m\in\mathbb{N}$ and $\epsilon>0$. Let

[TABLE]

(To avoid the excessive use of floors and ceilings, we assume that $J^{m}_{\epsilon}$ is a positive integer. The results extend easily to the general case.) Define the events

[TABLE]

We have that

[TABLE]

With the above equation in mind, we now proceed to demonstrate that both $)1-\mathbb{P}(\mathcal{A}^{m}_{\epsilon}))$ and ($1-\mathbb{P}(\mathcal{B}^{m}_{\epsilon}))$ converge to zero in the limit as $m\to\infty$. Note that $I_{mT}$ is a Poisson random variable with mean $m\beta_{m}T$. Using elementary tail bounds on the Poisson distribution, we have that

[TABLE]

which converges to zero as $m\to\infty$.

We next turn to the value of $\mathbb{P}(\mathcal{B}^{m}_{\epsilon})$. Recall from the definition of $Z^{m}_{k}[i]$ that

[TABLE]

It is therefore not difficult to verify that $\{\xi^{m}_{n}\}_{n\in\mathbb{N}}$ is a martingale, and our objective would be to derive an upper bound on its maximum upward excursion over $\{0,\ldots,J^{m}_{\epsilon}\}$. Unfortunately, we cannot apply the Azuma-Hoeffding inequality, because the $i$th increment of $\{\xi^{m}_{n}\}_{n\in\mathbb{N}}$ involves the term $\tau^{m}_{i}$, which does not admit a bounded support. Instead, we will use the following upper bound on the moment generating function of $\xi^{m}_{n}$, whose proof is based on Doob’s inequality and is given in Appendix D.7.

Lemma C.2

Fix $n\in\mathbb{N}$ and $\theta\in(0,m\beta_{m})$. We have that

[TABLE]

We are now ready to establish an upper bound on the quantity $1-\mathbb{P}(\mathcal{B}^{m}_{\epsilon})$. Recall that $J^{m}_{\epsilon}=(1+\epsilon)m\beta_{m}T$. Fix $\eta\in\left(0,\min\{T,1\}/2\right)$, and let $\theta_{0}=\eta\frac{\epsilon}{1+\epsilon}T^{-1}m\beta_{m}.$ In particular, $\theta_{0}\in(0,m\beta_{m}/2)$. We have that

[TABLE]

Step $(a)$ follows from Doob’s inequality and the fact that, for any $\theta>0$, the sequence $\left\{\exp\left(\theta\xi^{m}_{n}\right)\right\}_{n\in\mathbb{N}}$ is a submartingale, and $(b)$ from Lemma C.2 with $n=J^{m}_{\epsilon}$. Step $(c)$ is due to $\theta_{0}<m\beta_{m}/2$, and hence $m\beta_{m}-\theta_{0}>m\beta_{m}/2$. Finally, step $(d)$ follows from the definition of $\theta_{0}$ and that $\eta<1$.

With a derivation analogous to that of Eq. (172), we have that

[TABLE]

Therefore, combining Eq. (165) with Eqs. (166), (172) and (173), we have that

[TABLE]

Since $m\beta_{m}\to\infty$ as $m\to\infty$, the above equation further implies that

[TABLE]

We now bound the second term in Eq. (161). It is not difficult to verify, from Eq. (14), that there exists $l_{p}\in\mathbb{R}_{+}$ such that $p_{k}(\cdot)$ is $l_{p}$-Lipschitz continuous for all $k\in\mathcal{K}$. We have that

[TABLE]

It therefore suffices to show that, if $m\beta_{m}\to\infty$ as $m\to\infty$, then

[TABLE]

To this end, we have that

[TABLE]

In step $(a)$ we have invoked the property that $\widehat{Q}^{m}_{k}(\cdot)$ is piece-wise constant. Step $(b)$ follows from the definition of $G_{k}(\cdot,\cdot)$ in Eq. (37), and $(c)$ from the fact that $|V^{m}_{k}(t)|\leq q^{0}+\lambda_{k}t$ for all $t\in\mathbb{R}_{+}$ (Eq. (40)). It remains to derive an upper bound on the tail probabilities of the term $\sum_{i=1}^{I_{mT}+1}(\tau^{m}_{i})^{2}$, which is isolated in the form of the following lemma. The proof involves an elementary application of Markov’s inequality, and is given in Appendix D.8.

Lemma C.3

Suppose that $m\beta_{m}\to\infty$ as $m\to\infty$. We have that

[TABLE]

We are now ready to prove Eq. (177). By Eq. (178), we have that

[TABLE]

for all $\epsilon>0$, where the first inequality follows from a union bound, and the last step from Proposition 5.6 and Lemma C.3. Substituting Eqs. (175) and (180) into Eq. (161), we have that

[TABLE]

This establishes Eq. (154), which, in light of Eq. (153) completes the proof of Proposition 5.7.

C.3 Proof of Proposition 6.3

*Proof. *We begin by showing the following, strengthened version of Lemma 5.11, which states that a similar stochastic dominance property holds for $Q^{m}(\infty)$ even when conditioning on $C^{m}(\cdot)$ being of a specific value.

Lemma C.4

Let $W^{m}=(Q^{m},C^{m})$ be a random vector drawn from the steady-state distribution, $W^{m}(\infty)$. There exist constants , $m_{0}$ and $\gamma>0$, such that for all $m\geq m_{0}$,

[TABLE]

*Proof. *Let $\mathcal{Q}^{m}=\{{x}/{m}:\,x\in\mathbb{Z}_{+}^{\mathcal{K}}\}$. We have that, for all $k\in\mathcal{K}$, $n\in\mathbb{N}$,

[TABLE]

By Eq. (109) of Lemma 6.2, the above equation implies that

[TABLE]

where the last step follows from the fact that $\frac{\alpha_{0}}{K\alpha_{0}+\sum_{i\in\mathcal{K}}u_{i}}$ is always bounded from above by $1/K$.

By Eq. (86), we have that $U^{m}_{\lambda_{1}}/m\to\lambda_{1}$ almost surely as $m\to\infty$. Therefore, there exist $m_{0}$ and $y>0$, such that

[TABLE]

Combining Eq. (108) in Lemma 6.2 with Eqs. (184) and (185), we have that, for all $m\geq m^{0}$,

[TABLE]

where $\gamma\stackrel{{\scriptstyle\triangle}}{{=}}\frac{K\alpha_{0}+K(\lambda_{1}+y)}{\alpha_{0}\left(1-{1}/{2K}\right)}$. Step $(a)$ follows from Eq. (184), $(b)$ from Lemma 5.11 and a union bound, and $(c)$ from Eq. (185).

Fix $x\in\mathbb{Z}_{+}$. We have that, for all $m\geq m_{0}$,

[TABLE]

for all $i,k\in\mathcal{K}$, where step $(a)$ follows from Lemma 5.11, and $(b)$ from Eq. (186). Since the above inquality holds for all $x\in\mathbb{Z}_{+}$, this completes the proof of Lemma C.4. $\square$

We now prove the convergence in Eq. (112). Recall that the first update point, $S^{m}_{1}$, is exponentially distributed with mean $1/\beta_{m}$, and independent from $W^{m}(0)$. Define the event $\mathcal{E}^{m}=\{S^{m}_{1}\geq h(m)\}$. We have that

[TABLE]

Fix $i,k\in\mathcal{K}$. Recall from Eq. (86) that $U^{m}_{\lambda_{1}}/m$ converges to $\lambda_{1}$ almost surely as $m\to\infty$. This implies that there exists $\upsilon>0$, independent of $m$, such that

[TABLE]

where step $(a)$ follows from Lemma C.4.

Fix $x\in\mathbb{R}_{+}$. We have that

[TABLE]

where step $(a)$ follows from the independence between $\mathcal{E}^{m}$ and $C^{m}[0]$, and $(b)$ from Eq. (188).

We now bound the term on the right-hand side of Eq. (190). Denote by $B(n,p)$ a binomial random variable with $n$ trials and a success probability of $p$ per trial, and by $U^{m}_{\lambda}(t)$ the number of jobs in system at time $t$ in an initially empty $M/M/\infty$ queue with arrival rate $\lambda$ and departure rate $1/m$. We have that

[TABLE]

For step $(a)$, note that each unit of reward initially present at time $t=0$ has probability of $\exp(-S^{m}_{1}/m)$ or remaining in the system by $t=S^{m}_{1}$. Therefore, the rewards in site $i$ at time $S^{m}_{1}$ satisfy the following decomposition:

[TABLE]

The first term corresponds to the units of rewards at $t=S^{m}_{1}$ that had arrived during the interval $(0,S^{m}_{1})$, and hence is non-zero only if $i=k$. The second term corresponds to those individuals initially present at $t=0$ who remained in the system by $t=S^{m}_{1}$. Step $(b)$ follows from the definition of $\mathcal{E}^{m}$, and $(c)$ from the well-known fact that the number of jobs in system in an initially empty $M/M/\infty$ queue at any time is always stochastically dominated by its steady-state distribution.

Because $h(m)/m\to\infty$ as $m\to\infty$, we have that $\lim_{m\to\infty}\mathbb{E}\left(\frac{1}{m}B\left(\upsilon m,e^{-h(m)/m}\right)\right)=0$. Applying Markov’s inequality, we obtain that

[TABLE]

where $\stackrel{{\scriptstyle P}}{{\rightarrow}}$ denotes convergence in probability. Recall from Eq. (86) that, almost surely,

[TABLE]

Fix $\epsilon>0$, and substitute Eq. (191) into Eq. (190). We have that

[TABLE]

where step $(a)$ follows from Eq. (189), and $(b)$ from Eqs. (193) and (194). Using the same line of arguments as that in Eqs. (190) through (195), we can show that

[TABLE]

which, along with Eq. (195), yields that

[TABLE]

Since the above equation holds for all $i,k\in\mathcal{K}$, this proves Eq. (112) in Proposition 6.3.

We now turn to Eq. (113). Fix $i,k\in\mathcal{K}$. Using essentially identical arguments as those for Eq. (197), we can show that

[TABLE]

We have that,

[TABLE]

Step $(a)$ follows from a decomposition similar to Eq. (192), by the writing the recallable rewards at time $h(m)$ as those who arrived after $t=0$, which is dominated by $U^{m}_{\lambda_{1}}(h(m))$, and those who were in the system at $t=0$, which is dominated by $Q^{m}_{i}(0)|\{C^{m}[0]=k\}$. Step $(b)$ follows from Lemma 5.11. Since $U^{m}_{\lambda_{1}}$ is a Poisson distribution with mean $m\lambda_{1}$, it is not difficult show that there exists a random variable $Y\in\mathbb{R}_{+}$, such that

[TABLE]

Combining Eqs. (112) and (200), the dominated convergence theorem implies that, for all $i,k\in\mathcal{K}$,

[TABLE]

This shows Eq. (113), and thus completes the proof of Proposition 6.3. $\square$

C.4 Proof of Proposition 8.1

*Proof. *Fix $\eta<1$. Under the polynomial reward-matching model, the fluid solution satisfies

[TABLE]

setting the left-hand side to [math], we have that a state $q$ is an invariant state of the fluid solutions if

[TABLE]

We first show that the above equations admit a unique solution, $q^{I}$. That is, the fluid solutions admit a unique invariant state. Suppose, for the sake of contradiction, that there exist two distinct invariant states $q^{I}$ and $\tilde{q}^{I}$. Let

[TABLE]

and $\tilde{Z}=\sum_{i\in\mathcal{K}}(\tilde{q}^{I}_{i}\vee\alpha_{0})^{\eta}$ denote the denominators on the right-hand side of Eq. (203) under $q^{I}$ and $\tilde{q}^{I}$, respectively. From Eq. (203), by considering separately two cases depending on whether $q^{I}_{k}$ is smaller than $\alpha_{0}$, we have that the invariant state satisfies

[TABLE]

which indicates that if $Z=\tilde{Z}$, then $q^{I}=\tilde{q}^{I}$. Therefore, in order for $q^{I}$ and $\tilde{q}^{I}$ to be distinct, we must have that $\tilde{Z}\neq Z$. Without loss of generality, let us assume that $\tilde{Z}>Z$. Because $\eta<1$, by Eq. (205), we have that $q_{k}^{I}$ is a monotonically decreasing function of $Z$, for all $k$. We thus have that $\tilde{q}^{I}_{k}<q^{I}_{k}$ for all $k\in\mathcal{K}$. This leads to a contradiction, since when $\tilde{q}^{I}_{k}<q^{I}_{k}$ for all $k\in\mathcal{K}$, we will necessarily have that $\tilde{Z}$ is strictly less than $Z$. This proves that the solution to Eq. (203) must be unique.

We now find the unique invariant state $q^{I}$, and for now we assume that such $q^{I}$ exists. Note that when $\eta<1$, $q_{k}^{I}$ is a monotonically increasing function of $\lambda_{k}$ for $\lambda_{k}\geq 0$. Eq. (205) implies that $q_{i}\geq q_{j}$ if and only if $\lambda_{i}\geq\lambda_{j}$, which further implies that $q^{I}_{1}\geq\dots\geq q^{I}_{K}$. Since $q^{I}_{k}$ is non-increasing in $k$, we may define $i^{*}$ as the unique index such that

[TABLE]

where we define $i^{*}=0$ if $q^{I}_{1}<\alpha_{0}$, and $i^{*}=K$ if $q^{I}_{K}\geq\alpha_{0}$.

We now consider different values of $\alpha_{0}$. Suppose that $\alpha_{0}\geq\lambda_{1}/K$. It is not difficult to verify that in this case $i^{*}=0$, $q^{I}_{k}<\alpha_{0}$ for all $k\in\mathcal{K}$, and $Z=\sum_{i\in\mathcal{K}}(q^{I}_{i}\vee\alpha_{0})^{\eta}=K\alpha_{0}^{\eta}$. It follows from Eq. (205) that we must have $\lambda_{1}<Z\alpha^{1-\eta}=K\alpha_{0}$, or equivalently, $\alpha_{0}>\lambda_{1}/K$, and that

[TABLE]

This proves Item 1 in the proposition.

Consider next the other extreme where $\alpha_{0}$ is so small that $i^{*}=K$ and $q^{I}_{k}\geq\alpha_{0}$ for all $k$. By Eq. (205), this is to say that

[TABLE]

In this case, we have that $Z=\sum_{i\in\mathcal{K}}\left(\frac{\lambda_{i}}{Z}\right)^{\frac{\eta}{1-\eta}},$ which leads to, after rearrangement,

[TABLE]

Substituting the value of $Z$ from Eq. (209) into (208), we obtain the condition on $\alpha_{0}$:

[TABLE]

The expression of $q^{I}_{k}$ in Eq. (124) is obtained by substituting Eq. (209) into the top line of Eq. (205). This proves Item 2 of the proposition.

Finally, fix $\alpha_{0}$ such that $\lambda_{K}\frac{\lambda_{K}^{\frac{\eta}{1-\eta}}}{\sum_{i\in\mathcal{K}}\lambda_{i}^{\frac{\eta}{1-\eta}}}<\alpha_{0}\leq\lambda_{1}/K$. In this case, we have that $1\leq i^{*}\leq K-1$. By Eq. (205), we have that for all $k\geq i^{*}+1$,

[TABLE]

where the last equality follows from the fact that $q^{I}_{i^{*}}\geq\alpha_{0}$, and hence $q^{I}_{i^{*}}=\left(\frac{\lambda_{i^{*}}}{Z}\right)^{\frac{1}{1-\eta}}$. This yields

[TABLE]

which proves Eq. (130) in Item 3. It remains to identify the values of $i^{*}$ and $q^{I}_{i^{*}}$. By Eq. (205), in order to have $q^{I}_{i^{*}}\geq\alpha_{0}$, it is necessary and sufficient to have

[TABLE]

where the equalities $(a)$ and $(b)$ are based on Eq. (214), and inequality $(c)$ uses the fact that $q^{I}_{i^{*}}\geq\alpha_{0}$. Analogously, in order to have $q^{I}_{i^{*}}<\alpha_{0}$, it is necessary and sufficient to have

[TABLE]

where inequality $(a)$ is derived from the bottom line of Eq. (214):

[TABLE]

Eqs. (218) and (222) thus give a charaterization of $i^{*}$ in terms of the problem primitives, and establish Eq. (126). Note that the proceeding analysis has already shown that $i^{*}$ is unique and lies in $\{1,\ldots,K-1\}$, although the uniqueness can also be derived easily by noticing that $g(\cdot)$ is a non-decreasing function (since $\left(\frac{\lambda_{j}}{\lambda_{i}}\right)^{\frac{\eta}{1-\eta}}\geq 1$ whenever $j\leq i$) and the $\lambda_{i}$’s are non-increasing in $i$.

Finally, we show that $q^{I}_{i^{*}}$ exists and is given by the unique solution to Eq. (127). Substituting the expressions for the $q^{I}_{k}$’s from Eq. (214) into Eq. (203) leads to Eq. (127). In particular, $q^{I}_{i^{*}}$ is the solution to the following equation:

[TABLE]

To see that such a solution exists and is unique for any fixed $i^{*}\in\{1,\ldots,K-1\}$, note that since $\eta<1$, the left-hand side of the above equation is a strictly increasing function in $q^{I}_{i^{*}}$, which grows from [math] to $\infty$ as $q^{I}_{i^{*}}$ varies from [math] to $\infty$; the right-hand side, on the other hand, is a strictly decreasing function in $q^{I}_{i^{*}}$, which, as $q^{I}_{i^{*}}$ varies from [math] to $\infty$, decreases from $\frac{\lambda_{i^{*}}}{(K-i^{*})\alpha_{0}^{\eta}}$ to [math]. Together, they imply that Eq. (223) must admit a unique solution in $\mathbb{R}_{+}$. This completes the proof of Proposition 8.1. $\square$

Appendix D Proofs for Lemmas

D.1 Proof of Lemma 5.2

*Proof. *The existence and uniqueness of the fluid solution follow from Picard’s existence theorem (Section 2, Chapter 1, Coddington and Levinson, (1955)) by verifying that the right-hand side of Eq. (15) is uniformly Lipschitz-continuous in $q(t)$ over $\mathbb{R}_{+}^{\mathcal{K}}$, and (trivially) continuous in $t$. To show the fluid solution’s continuous dependence on initial condition, note that because of the Lipschitz continuity of $p_{k}(\cdot)$, there exists a constant $l>0$, such that for initial conditions $x,y\in\mathbb{R}_{+}^{\mathcal{K}}$ and $t\in\mathbb{R}_{+}$, we have that

[TABLE]

where the last inequality follows from Gronwall’s lemma (Proposition B.2). Therefore, $\lim_{x\to y}q(x,t)=q(y,t)$ for all $y\in\mathbb{R}_{+}^{\mathcal{K}}$. This completes the proof of the lemma. $\square$

D.2 Proof of Lemma 5.11

*Proof. *We will use a simple coupling argument as follows. Fixing $k\in\mathcal{K}$, the evolution of the process $U^{m}_{\lambda_{k}}(\cdot)$ corresponds to that of $Q^{m}_{k}(\cdot)$ if action $k$ were selected for all $t\in\mathbb{R}_{+}$, and it follows, given the same initial condition, that $Q^{m}_{k}(t)$ is stochastically dominated by $U^{m}_{k}(t)$ for all $t\in\mathbb{R}_{+}$. Since $U^{m}_{k}(\cdot)$ is positive recurrent for all $k\in\mathcal{K}$, we know that $Q^{m}(\cdot)$ is also positive recurrent, which in turn implies the positive recurrence of $\overline{W}^{m}(\cdot)$, because the evolution of $C^{m}(\cdot)$ is derived by sampling from $\mathcal{K}$ based solely on the value of $Q^{m}(\cdot)$. Lemma 5.11 follows form the above-mentioned stochastic dominance, the fact that under any finite initial condition, $U^{m}_{k}(t)$ converges in distribution to $U^{m}_{\lambda_{k}}$ as $t\to\infty$, and the observation that $U^{m}_{\lambda^{\prime}}\preceq U^{m}_{\lambda}$ whenever $\lambda\geq\lambda^{\prime}\geq 0$. $\square$

D.3 Proof of Lemma 5.12

*Proof. *To show Eq. (89), observe that

[TABLE]

where step $(a)$ follows from Lemma 5.11 and the union bound, and $(b)$ from the elementary inequality $i!\geq(i/e)^{i}$. Eq. (225) thus shows that for all $\epsilon$, $\inf_{m\in\mathbb{N}}\pi_{W}^{m}(\overline{\mathcal{Q}}_{x})>1-\epsilon$ for all sufficiently large $x$, which proves Eq. (89). $\square$

D.4 Proof of Lemma 5.13

*Proof. *We first show the following uniform convergence property:

[TABLE]

where $\overline{Q}^{m}(x,\cdot)$ denotes a process $\overline{Q}^{m}(\cdot)$ initialized with $\mathbb{P}(\overline{Q}^{m}(0)=x)=1$. Suppose, for the sake of contradiction, that there exist $\delta>0$ and a sequence $\{x_{i}\}_{i\in\mathbb{N}}$, $x_{i}\in S\cap\mathcal{Q}^{m}$, such that

[TABLE]

Because $S$ is compact, there exists a sub-sequence $\{x_{i_{j}}\}_{j\in\mathbb{N}}\subset\{x_{i}\}_{i\in\mathbb{N}}$ and $x^{*}\in S$ such that $x_{i_{j}}\to x^{*}$ as $j\to\infty$. We have that

[TABLE]

where step $(a)$ follows from the fact that, for all $t\in\mathbb{R}_{+}$, $q(x,t)$ is continuous with respect to $x$ (Lemma 5.2). This leads to a contradiction with Theorem 5.5, and hence proves Eq. (226).

D.5 Proof of Lemma 6.2

*Proof. *It is not difficult verify that $\{W^{m}[n]\}_{n\in\mathbb{Z}_{+}}$ is a time-homogeneous, aperiodic, and irreducible Markov chain. Because the continuous-time process, $\{W^{m}(t)\}_{t\in\mathbb{R}_{+}}$, is positive recurrent, so is its discrete-time counterpart, $\{W^{m}[n]\}_{n\in\mathbb{N}}$, and $W^{m}[n]$ converges to its steady-state distribution, $W^{m}[\infty]$, as $n\to\infty$. Eq. (108) follows from the same argument as that of Lemma 5.11. We now show Eq. (109). Denote by $N^{m}(t)$ the index of the last update point by time $t$:

[TABLE]

and by $T^{m}(t)$ its value

[TABLE]

Recall that the update points are generated according to a Poisson process, and it not difficult to show that, almost surely, $N^{m}(t)\to\infty$ and $T^{m}(t)\to\infty$, as $t\to\infty$. We thus have that, for all $k\in\mathcal{K}$,

[TABLE]

The same argument applies for $Q^{m}[\infty]$ versus $Q^{m}(\cdot)$. This completes the proof of Lemma 6.2. $\square$

D.6 Proof of Lemma C.1

*Proof. *Define

[TABLE]

Let $\left\{\mathcal{F}_{t}\right\}_{t\in\mathbb{R}_{+}}$ be the natural filtration associated with $W^{m}(\cdot)$. It is not difficult to show, by the definition of $N(\cdot)$, that $\{\gamma(t)\}_{t\in\mathbb{R}_{+}}$ is martingale with respect to $\{\mathcal{F}_{t}\}_{t\in\mathbb{R}_{+}}$. Define the stopping time

[TABLE]

Let $\{\tilde{N}(t)\}_{t\in\mathbb{N}}$ be a counting process defined as:

[TABLE]

That is, $\tilde{N}(t)$ coincides with $N(t)$ up until $t=T_{s}$, and stays constant afterwards. Let

[TABLE]

Then, it is not difficult to show that $\tilde{N}(\cdot)$ is a counting process whose instantaneous rate at time $t$ is $\tilde{\psi}(t)$, and the process

[TABLE]

is a martingale with respect to $\{\mathcal{F}_{t}\}_{t\in\mathbb{R}_{+}}$. From the definition of $\tilde{\gamma}(\cdot)$ and $\gamma(\cdot)$, we have that, for all $\epsilon>0$,

[TABLE]

To complete the proof, therefore, it suffices to show that

[TABLE]

We now show Eq. (238) using Doob’s inequality (Proposition B.1 in Appendix B), by following a line of arguments similar to that used in the proof of Theorem 2.2 of Kurtz, (1978). First, we introduce a representation of the Markov process $W^{m}(\cdot)$ using Poisson processes. Let $\{\Xi_{y}(\cdot)\}_{y\in\mathbb{Z}^{|\mathcal{K}|+1}}$ a family of mutually independent unit-rate Poisson counting processes, indexed by $\mathbb{Z}^{|\mathcal{K}|+1}$. For every $y\in\Xi_{y}$, let $\psi_{y}(\cdot)$ be the rate function of $W^{m}(\cdot)$ for the jump with value $y$, i.e., $\psi_{y}(w)$ is the instantaneous rate at which $W^{m}(\cdot)$ jumps to state $w+y$ when in state $w\in\mathbb{Z}_{+}^{K+1}$. Then, the process $W^{m}(\cdot)$ can be expressed as a solution to the following integral equation:

[TABLE]

In this representation, $\Xi_{y}\left(\int_{0}^{t}\psi_{y}(W^{m}(s))ds\right)$ counts the number of jumps of value $y$ over the interval $[0,t]$. We thus have that, by setting $y=l$,

[TABLE]

and

[TABLE]

Fix $\theta>0$. Since $\tilde{\gamma}(\cdot)$ is a martingale and $\exp(\cdot)$ a positive convex function, $\{\exp(\theta\tilde{\gamma}(t))\})_{t\in\mathbb{R}_{+}}$ is a submartingale. From Eq. (241), we have that

[TABLE]

where $\tau_{t}=\int_{0}^{t}\tilde{\psi}(s)ds$. Note that $\tau_{T}$ is a stopping time with respect to $\Xi_{l}(\cdot)$. Since by definition $\tilde{\psi}(t)\leq\phi$ for all $t$, we have that $\tau_{T}\leq\phi T$. Applying the optional sampling theorem for submartingales indexed by partially ordered sets (cf. Washburn and Willsky, (1981)) to $\left\{\exp(\theta\tilde{\gamma}(t))\right\}_{t\in\mathbb{R}_{+}}$, we have that

[TABLE]

where the last inequality follows from the fact that $\Xi_{l}(\phi T)$ is a Poisson random variable with mean $\phi T$ whose moment generating function is given by $\mathbb{E}(\exp(a\Xi_{l}(\phi T)))=\exp(\phi T(e^{a}-1))$, $a\in\mathbb{R}$. Analogously, we can show that

[TABLE]

where the last inequality follows from the fact that $e^{\theta}-e^{-\theta}\geq 2\theta$ for all $\theta\geq 0$.

Note that for any $\theta>0$, both $\left\{\exp\left(\theta\gamma(t)\right)\right\}_{t\geq 0}$ and $\left\{\exp\left(-\theta\gamma(t)\right)\right\}_{t\geq 0}$ are non-negative submartingales. Using Doob’s inequality and Eqs. (243) and Eq. (244), we have that, for all $\epsilon,\theta>0$,

[TABLE]

By setting $\theta=\log\left(1+{\epsilon}/{\phi T}\right)$ in Eq. (246), we conclude that

[TABLE]

where $h(x)\stackrel{{\scriptstyle\triangle}}{{=}}(1+x)\log(1+x)-x$. This completes the proof. $\square$

D.7 Proof of Lemma C.2

*Proof. *We show the result by induction. For the base case, we extend the definition of $\{\xi^{m}_{i}\}_{i\in\mathbb{N}}$ by letting $\xi^{m}_{0}\stackrel{{\scriptstyle\triangle}}{{=}}0$, and it is not difficult to see that the inequality holds when $n=0$.

Fix $i\in\{0,\ldots,n-1\}$, and suppose that

[TABLE]

In light of the base case, it then suffices to show that the above equation implies

[TABLE]

Let $\{\mathcal{C}_{l}\}_{l\in\mathbb{Z}_{+}}$ be the natural filtration induced by $\{\tau_{l}^{m},Z^{m}_{k}[l],\overline{Q}^{m}[l]\}_{l\in\mathbb{N}}$, with $\mathcal{C}_{0}\stackrel{{\scriptstyle\triangle}}{{=}}\emptyset$. We have that

[TABLE]

where step $(a)$ follows from $\xi^{m}_{i}$ being $\mathcal{C}_{i}$-measurable. We now develop an upper bound for the second term on the right-hand side of Eq. (250), as follows.

[TABLE]

Step $(a)$ follows from the fact that, for a given value of $\overline{Q}^{m}[i+1]$, $Z^{m}_{k}[i+1]$ is a Bernoulli random variable with $\mathbb{P}\left(Z^{m}_{k}[i+1]=1\right)=p_{k}(\overline{Q}^{m}[i+1])$, and is independent from $\tau^{m}_{i+1}$. For step $(b)$, note that $\tau_{i+1}$ is an exponential random variable with mean $(m\beta_{m})^{-1}$, independent from $\mathcal{C}_{i}$ and $\overline{Q}^{m}[i+1]$. Its moment generating function is given by $\mathbb{E}\left(e^{h\tau_{i+1}}\right)=\frac{\beta_{m}}{\beta_{m}-h}$ for all $h\leq\beta_{m}$, where $h$, in our case, corresponds to $\theta(1-p_{k}(\overline{Q}^{m}[i+1]))$ and $\theta p_{k}(\overline{Q}^{m}[i+1])$, for the two terms respectively. Step $(c)$ stems from the fact that $1+x\leq e^{x}$ for all $x\in\mathbb{R}_{+}$. Finally, for step $(d)$ we have used the fact that $p_{k}(\overline{Q}^{m}[i+1])\in[0,1]$ by definition, and that $\theta<m\beta_{m}$; the exponent is hence bounded from above by setting $p_{k}(\overline{Q}^{m}[i+1])$ to $1/2$ and [math] in the numerator and denominator, respectively.

Substituting Eq. (258) into Eq. (250), and invoking the induction hypothesis of Eq. (247), we have that

[TABLE]

This proves our claim. $\square$

D.8 Proof of Lemma C.3

*Proof. *Fix $\epsilon>0$ and $m\in\mathbb{N}$. We have that

[TABLE]

where step $(a)$ follows from Eq. (166), $(b)$ from the Markov’s inequality, and $(c)$ from $\tau^{m}_{i}$ being an exponentially distributed random variable with mean $(m\beta_{m})^{-1}$ and hence $\mathbb{E}((\tau^{m}_{1})^{2})=2(m\beta_{m})^{-2}$. Because $m\beta_{m}\to\infty$ as $m\to\infty$, the claim follows. $\square$