Time Complexity Analysis of a Distributed Stochastic Optimization in a Non-Stationary Environment

TL;DR

This paper analyzes the time complexity of a distributed stochastic optimization algorithm in non-stationary environments, demonstrating near-optimal performance with high probability and almost sure convergence, supported by wireless sensor network simulations.

Contribution

It introduces an approximate distributed Drift-Plus-Penalty algorithm for non-stationary settings, providing convergence guarantees and performance bounds.

Findings

Achieves near-optimal time average cost within epsilon of the optimum.

Proves almost sure convergence of the algorithm despite delays in common information.

Validates theoretical results through wireless sensor network simulations.

Abstract

In this paper, we consider a distributed stochastic optimization problem where the goal is to minimize the time average of a cost function subject to a set of constraints on the time averages of related stochastic processes called penalties. We assume that the state of the system is evolving in an independent and non-stationary fashion and the "common information" available at each node is distributed and delayed. Such stochastic optimization is an integral part of many important problems in wireless networks such as scheduling, routing, resource allocation and crowd sensing. We propose an approximate distributed Drift- Plus-Penalty (DPP) algorithm, and show that it achieves a time average cost (and penalties) that is within epsilon > 0 of the optimal cost (and constraints) with high probability. Also, we provide a condition on the convergence time t for this result to hold. In particular, for any delay D >= 0 in the common information, we use a coupling argument to prove that the proposed algorithm converges almost surely to the optimal solution. We use an application from wireless sensor network to corroborate our theoretical findings through simulation results.

Equations

$$u_0(t)\triangleq \min \left\{\frac{{\alpha }_1(t){\omega }_1(t)}{3}+\frac{{\alpha }_2(t){\omega }_2(t)+{\alpha }_3(t){\omega }_3(t)}{6},1\right\},$$

$${\alpha }_i(t)\triangleq f_i({\omega }_i(t),Y_c(t)),i=1,2,\dots ,N,$$

$$\underset{\alpha (\tau )\in \mathcal{A}:\tau \in \mathbb{N}}{\min }$$

$$\underset{t\to \infty }{\lim }\Vert {\pi }_t-\pi {\Vert }_1=0.$$

$${\beta }_{\text{ALG},k}(s,\alpha |\mathcal{E})\triangleq \underset{t\in \mathbb{N},t\ge \alpha }{\sup }\Vert {\mathbb{M}}_{t,s,k}(\mathcal{E}){\Vert }_{\text{TV}},$$

$$r_{k,{\pi }^{{}^{\prime }}}^{(m)}$$

$$\mathbb{E}{p}_k(t)$$

$$\underset{{\theta }_1,{\theta }_2,\dots ,{\theta }_F}{\min }$$

$${\mathcal{P}}_{i^{\ast }}\triangleq \arg \underset{\mathcal{Q}\in \{{\mathcal{P}}_1,\dots ,{\mathcal{P}}_{M_{\delta }}\}}{\min }\Vert \pi -\mathcal{Q}{\Vert }_1,$$

$$\underset{{\theta }_1,{\theta }_2,\dots ,{\theta }_F}{\min }$$

$$G(x)\triangleq \inf \left\{\sum _{m=1}^F{\theta }_m{r}_{0,{\mathcal{P}}_{i^{\ast }}}^{(m)}:\Theta \in {\mathcal{C}}_{x,\Theta }\right\},$$

$${\mathcal{C}}_{x,\Theta }\triangleq$$

$$\left\{\Theta :\sum _{m=1}^F{\theta }_m{r}_{k,{\mathcal{P}}_{i^{\ast }}}^{(m)}\le c_k+x,k=1,2,\dots ,K,\Theta 1^T=1\right\},$$

$$\left|G(x)-G(y)\right|\le c\left|x-y\right|,\text{ for all }x,y\ge 0.$$

$$Q_k(t+1)=\max \{Q_k(t)+p_k(t-D)-c_k,0\},$$

$$\mathcal{L}(t)\triangleq \frac{1}{2}\Vert {\mathbf{Q}}_t{\Vert }_2^2=\frac{1}{2}\sum _{i=1}^K{Q}_i^2(t),$$

$$\mathbb{E}\left[\Delta (t+D)+V{p}_0(t)|{\mathbf{Q}}_t\right]\le B_t(1+2D)+$$

$$V\sum _{m=1}^F{\beta }_m(t)r_{0,{\pi }_t}^{(m)}+\sum _{k=1}^K{Q}_k(t){\mathcal{C}}_{i,k,t},$$

$$B_t\triangleq \underset{m\in \{1,2,\dots ,F\}}{\max }\frac{1}{2}\sum _{k=1}^K\sum _{\omega \in \Omega }{\pi }_t(\omega ){\left|p_k({\mathbf{S}}^m(\omega ),\omega )-c_k\right|}^2,$$

$$j_t^{\ast }\triangleq \arg \underset{j\in \{1,2,\dots ,M_{\delta }\}}{\max }\frac{1}{w_t}\sum _{\tau =t-D-w_t+1}^{t-D}\log \left({\mathcal{P}}_j(\omega (\tau ))\right).$$

$$V{r}_{0,{\mathcal{P}}_{j_t^{\ast }}}^{(m_t)}+\sum _{k=1}^K{Q}_k(t)r_{k,{\mathcal{P}}_{j_t^{\ast }}}^{(m_t)}.$$

$$\mathrm{Pr}\left\{\frac{1}{t}\sum _{\tau =0}^{t-1}{p}_k(\tau )-c_k>{ϵ}_k\right\}$$

$$\Pr\{\mathcal{E}_{\delta,\tau}\}\leq P^{(\tau)}_{e,\texttt{up}}\triangleq\left\{\begin{array}[]{cc}q^{(\tau)}_{e,\texttt{up}}&\text{ if }\tau>D+w_{\tau}-1,\\ \frac{1}{M_{\delta}}&\text{otherwise},\end{array}\right.$$

$${\mathcal{D}}_{\tau ,j}\triangleq \frac{1}{w_{\tau }}\sum _{s=\tau -D-w_{\tau }+1}^{\tau -D}{\mathbb{E}}_{{\pi }_{\tau }}\log \left(\frac{{\mathcal{P}}_j(\omega (s))}{{\mathcal{P}}_{i^{\ast }}(\omega (s))}\right),$$

$$\frac{1}{t}\sum _{\tau =0}^{t-1}{p}_0(\tau )\le$$

$$\frac{1}{t}\sum _{\tau =0}^{t-1}{p}_k(\tau )\le c_k+Q_{\text{up}}(t)+\frac{{\alpha }_t(p_{\text{max},k}-p_{\text{min},k})}{t-{\alpha }_t}+ϵ,$$

$${\beta }_1^{\ast }\triangleq (t-{\alpha }_t)\left[\underset{k\ne 0}{\max }{\beta }_{\text{ADPP},k}(u_t,{\alpha }_t|{\mathcal{E}}_{[{\alpha }_t:t]})+S_{t,\delta }\right].$$

$${\mathcal{T}}_{t,i}\triangleq \left\{t:(t-{\alpha }_t)>\frac{(\Delta p{)}_{\text{max},0}{u}_t}{\sqrt{2}ϵ}\sqrt{\log \left(\frac{u_t}{{\gamma }_i-{\beta }_i^{\ast }}\right)}\right\},$$

$${\psi }_t(\delta )\triangleq$$

$${\beta }_{\text{ADPP},k}(s,{\alpha }_t|{\mathcal{E}}_{[{\alpha }_t:t]}^c)\le \underset{t\ge {\alpha }_t}{\sup }\sqrt{\frac{I(X_{k,t};X_{k,t-s}|{\mathcal{E}}_{[{\alpha }_t:t]}^c)}{2}},$$

$$I({\mathbf{X}}_t;{\mathbf{X}}_{t-s}|{\mathcal{E}}_{[{\alpha }_t:t]}^c)$$

Full text

Index terms: Drift-plus-penalty, Lyapunov function, wireless networks, online learning, distributed stochastic optimization.

I Introduction

Stochastic optimization is ubiquitous in various domains such as communications, signal processing, power grids, inventory control for product assembly systems and dynamic wireless networks [1, 2, 3, 4, 5, 6, 7, 8, 9]. A typical stochastic optimization problem involves designing control action for a given state of the system that minimizes the time average of a cost function subject to a set of constraints on the time average penalties [1, 2]. Both cost and penalties depend on the state of the system and the control actions taken by the users. For example, in a typical wireless application, the cost function refers to the instantaneous rate, and the penalty refers to the instantaneous power consumed. Further, the state here refers to the channel conditions. An algorithm known as Drift-Plus-Penalty (DPP) (see [10, 11, 12, 13, 14]) is known to provide a solution for these problems with theoretical guarantees. At each time slot, the DPP method, an extension of the back-pressure algorithm [15, 16], finds a control action that minimizes a linear combination of the cost and the drift. In the problem that we consider, the drift is a measure of the deviation (of the penalties) from the constraints, and the penalty corresponds to the cost. The DPP algorithm is shown to achieve an approximately optimal solution even when the system evolves in a non-stationary fashion, and is robust to non-ergodic changes in the state [10].

The DPP algorithm mentioned above assumes that the control action is taken at a centralized unit where the complete state information is available. However, wireless network and crowd sensing applications require a distributed control action that uses only the delayed state information at each node [17, 10]. This calls for a distributed version of the DPP algorithm with theoretical guarantees. The author in [4] considers a relaxed version of the above problem. In particular, assuming i.i.d. states with correlated “common information,” the author in [4] proposes a distributed DPP algorithm, and proves that the approximate distributed DPP algorithm is close to being optimal. Several authors use the above results in various contexts such as crowd sensing [17], energy efficient scheduling in MIMO systems [18], to name a few. However, in many practical applications, the states evolve in a dependent and non-stationary fashion [13]. Thus, the following assumptions about the state made in [4] need to be relaxed: (i) independent and (ii) identically distributed. Further, from practical and theoretical standpoints, it is important to investigate the rate of convergence of the distributed algorithm to the optimal. In this paper, we relax the assumption (ii) above, and unlike [4], we provide a Probably Approximately Correct (PAC) bound on the performance. Also, we prove an almost sure convergence of the proposed distributed algorithm to a constant within the optimal. We would like to emphasize that extending the analysis in [4] to non-stationary states is non-trivial. The only work that provides a “PAC type” result for the DPP algorithm is [19]. However, the authors consider i.i.d. states, and the decision is centralized. Moreover, the method used in [19] cannot be directly extended to a problem with non-stationary states since their proof requires the control action to be stationary, and this assumption in general is not true. Now, we highlight the contribution of our work.

I-A Main Contribution of the Paper

In this paper, we consider a distributed stochastic optimization problem when the states evolve in an independent and non-stationary fashion. In particular, we assume that the state is asymptotically stationary, i.e., the probability measure $\pi_{t}$ of the state $\omega(t)\in\Omega$ converges to a probability measure $\pi$ as $t\rightarrow\infty$ in the $\mathcal{L}_{1}$-norm sense. This assumption makes the extension of the method in [4] non-trivial. When $\pi_{t}=\pi$ for all $t\in\mathbb{N}$, the author in [4] proves theoretical guarantees by making use of the equivalence between a Linear Program (LP) that is a function of $\pi$ and the original stochastic optimization problem. However, when the probabilities are changing, this equivalence is difficult to establish. Instead, we show that the original problem is equivalent to a “perturbed” LP, which is a function of the limiting distribution $\pi$. Under mild conditions, we prove that the solution to the perturbed LP is approximately equal to that of the original problem. We use this result to prove theoretical guarantees for an approximate DPP algorithm that we propose in the paper. Moreover, unlike the previous works, we are more interested in providing sample complexity bounds rather than just dealing with the averages. The following are the main contributions of our work

For the above model, we show that with high probability, the average cost and penalties obtained by using the proposed approximate distributed DPP are within constants of the optimal solution and the constraints, respectively, provided the waiting time $t>\text{a threshold}$ (see Theorem 3). The threshold and the constants capture the degree of non-stationarity (i.e., $\|\pi_{t}-\pi\|_{1}$), and the number of samples used to compute an estimate of the state distribution. 2. 2.

Using the high probability result, we show that the cost corresponding to the proposed algorithm almost surely converges to a constant within $\epsilon_{0}>0$ of the optimal cost. We also show that the penalties induced by the proposed algorithm are within constants of the constraint values almost surely. It turns out that although the states are independent, the proposed algorithm induces dependencies across time in the cost and penalties. To prove the PAC and the almost sure convergence results, we use a coupling argument where the dependent sequence of the cost (also, penalties) is replaced by an independent sequence which results in an error expressed in terms of the $\beta_{1}$-mixing coefficient; a term that captures the stochastic dependency across time (see Sec. II). The $\beta_{1}$-mixing coefficient is bounded using information theoretic techniques to complete the proof. 3. 3.

We show that due to non-stationarity of the states, the performance gap goes down slowly compared to i.i.d. states. This is captured through $\|\pi_{t}-\pi\|_{1}$ and a term that depends on the measure of the complexity of the probability space averaged with respect to $\pi_{t}$ (see Theorem 3). Finally, we provide simulation results of a sensor network application, which is a particular use case scenario of the problem considered.

The paper is organized as follows. The problem statement, an approximate DPP Algorithm with related theoretical guarantees and simulation results are provided in Sec. II, Sec. III and Sec. V, respectively. A bound on the mixing coefficient is provided in Sec. IV. Sec. VI concludes the paper.

Notation: We use the following notations in the paper. We write $f(x)\doteq g(x)$, $f(x)\preceq g(x)$, $f(x)\prec g(x)$, $f(x)\succeq g(x)$, and $f(x)\succ g(x)$ to mean $\lim_{x\rightarrow\infty}\frac{f(x)}{g(x)}=1$, $\lim_{x\rightarrow\infty}\frac{f(x)}{g(x)}\leq 1$, $\lim_{x\rightarrow\infty}\frac{f(x)}{g(x)}<1$, $\lim_{x\rightarrow\infty}\frac{f(x)}{g(x)}\geq 1$ and $\lim_{x\rightarrow\infty}\frac{f(x)}{g(x)}>1$, respectively. We use $f(x)=\mathcal{O}(g(x))$ if $\lim_{x\rightarrow\infty}\frac{f(x)}{g(x)}=c$ for some $c<\infty$.

II Motivation and Problem Statement

Towards motivating the system model studied in the paper, we consider a network of $3$ sensors, where the sensor $i$ observes the state $\omega_{i}(t)\in\{0,1,2,3\}$, $i=1,2,3$, and reports the observation to a central unit [4]. The reporting incurs a penalty in terms of the power consumed by the sensors to transmit the state information. The state $\mathbf{\omega}{(t)}\triangleq\{\omega_{1}(t),\omega_{2}(t),\omega_{3}(t)\}$, $t\in\mathbb{N}$ in general is a stochastic process that evolves in a non-stationary fashion. Assume that the central unit trusts sensor $1$ more than the others. The problem is to maximize the average of the following utility function subject to the constraint that the average power consumed by each sensor is less than $\bar{P}$:

[TABLE]

where $\alpha_{i}(t)\in\{0,1\}$, $i=1,2,3$ are the decision variables. Note that if $\omega_{i}(t)=3$ for $i=1,2,3$, and $\alpha_{i}(t)=1$ for $i=2,3$, then there is no increase in the utility if sensor $1$ also decides to transmit, i.e., $\alpha_{1}(t)=1$. However, none of the sensors know the entire state of the system. In this case, the sensor $1$ may also choose to transmit, thus wasting its power leading to a suboptimal operation compared to a centralized scheme. In order to resolve this issue in a distributed setting, we assume that a delayed “common information” is available (see Sec. II of [4] for more details) using which each sensor picks one of the “pure strategies”. For example, each sensor can acquire the information about the state $\omega(t)$ with a fixed delay $D>0$. In this case, the “common information” can be some function of $\omega(t-D)$. Thus, the problem is to find the set of optimal decision variables in a distributed fashion with “common information” that maximizes the average of the above utility subject to the constraints on the average power. Next, we describe the system model that generalizes the above example, and later provide an algorithm with theoretical guarantees.

Consider a system comprising of $N$ users making decisions in a distributed fashion at discrete time steps $t\in\{0,1,2,\ldots\}$. Each user $i$ observes a random state $\omega_{i}(t)\in\Omega_{i}$, and a “common information” $Y_{c}(t)\in\mathcal{Y}$ to make a control decision $\alpha_{i}(t)\in\mathcal{A}_{i}$, $i=1,2,\ldots,N$. Here, for each user $i$, $\Omega_{i}$, $\mathcal{Y}$ and $\mathcal{A}_{i}$ denote the state space, common information space and action/control space, respectively. Let ${\omega}(t)\triangleq\{\omega_{1}(t),\omega_{2}(t),\ldots,\omega_{N}(t)\}\in\Omega$ and $\alpha(t)\triangleq\{\alpha_{1}(t),\alpha_{2}(t),\ldots,\alpha_{N}(t)\}\in\mathcal{A}$, where $\Omega\triangleq\Omega_{1}\times\Omega_{2}\times\ldots\times\Omega_{N}$, and $\mathcal{A}\triangleq\mathcal{A}_{1}\times\mathcal{A}_{2}\times\ldots\times\mathcal{A}_{N}$. Also, let us assume that the number of possible values that $p_{k}(t)$ takes is finite and equal to $\mu_{k}\in\mathbb{N}$, $k=1,\ldots,K$. The decision is said to be distributed if (see [4])

• •

There exists a function $f_{i}:\Omega_{i}\times\mathcal{Y}\rightarrow\mathcal{A}_{i}$, such that

[TABLE]

where $Y_{c}(t)$ belongs to the common information set $\mathcal{Y}$.

• •

The common information $Y_{c}(t)$ is independent of $\omega(t)$ for every $t\in\mathbb{N}$.

At each time slot $t$, the decision $\alpha(t)$ and the state $\omega(t)$ result in a cost $p_{0}(t)\triangleq p_{0}(\alpha(t),\omega(t))$ and penalties $p_{k}(t)\triangleq p_{k}(\alpha(t),\omega(t))$, $k=1,2,\ldots,K$. The central goal of the paper is to analyze an approximate distributed solution to the following problem when $\omega(t)$, $t\in\mathbb{N}$ is independent and non-stationary, $\mathbf{P_{0}:}$

[TABLE]

In the above, the expectation is jointly with respect to the distribution of the state $\omega(t)$ and a possible randomness in the decision $\alpha(t)$, $t\in\mathbb{N}$. Let $p^{(opt)}$ be the optimal cost corresponding to the problem $\mathbf{P_{0}}$. Note that the first equation in $\mathbf{P_{0}}$ represents the time average cost while the second and the third equations represent constraints on the penalties and the decisions, respectively. Informally, we are interested in proving a Probably Approximately Correct (PAC) type result of the following form [19]

• •

For every $\epsilon_{k}>0$, with a probability of at least $1-\delta_{k}$, $\frac{1}{t}\sum_{\tau=0}^{t-1}p_{k}^{(\approx)}(\tau)\leq c_{k}+\epsilon_{k}$ provided $t>$ a threshold, where $p_{0}^{(\approx)}(\tau)$ and $p_{k}^{(\approx)}(\tau)$, $k=1,2,\ldots,K$ are the cost and penalties, respectively, of an approximate distributed scheme at $\tau\in\mathbb{N}$. Here $c_{0}\triangleq p^{(\textit{opt})}$ is the optimal cost, and $c_{k}$, $k=1,2,\ldots,K$ are as defined in $\mathbf{P_{0}}$.

First, unlike the model in [4], we assume that the state $\omega(t)$ evolves in an independent and non-stationary fashion across time $t$. In particular, the distribution of $\omega(t)$ denoted $\pi_{t}(\omega)$, $\omega\in\Omega$ satisfies the following asymptotic stationarity property.

Assumption 1: Assume that there exists a probability measure $\pi(\omega)$ on $\Omega$ such that

[TABLE]

Note that the efficacy of the distributed algorithm depends on how accurately each node computes an estimate of $\pi_{t}$, $t\in\mathbb{N}$. Naturally, we expect the bounds that we derive to be a function of the complexity of the probability measure space from which the “nature” chooses $\pi_{t}(\omega)$. Let us assume that for each $t\in\mathbb{N}$, $\pi_{t}$ is chosen from a set $\mathcal{P}$. Assuming that $\mathcal{P}$ is a closed set with respect to the $\mathcal{L}_{1}$-norm, we have $\pi\in\mathcal{P}$. One way of measuring the complexity is through the covering number, and the metric entropy of the set $\mathcal{P}$, which are defined as follows.

**Definition 1: **(see [20]) A $\delta$-covering of $\mathcal{P}$ is a set $\mathcal{P}_{c}\triangleq\{\mathcal{P}_{1},\mathcal{P}_{2},\ldots,\mathcal{P}_{M}\}\subseteq\mathcal{P}$ such that for all $\pi^{{}^{\prime}}\in\mathcal{P}$, there exists a $\mathcal{P}_{i}\in\mathcal{P}_{c}$ for some $i=1,2,\ldots,M$ such that $\|\pi^{{}^{\prime}}-\mathcal{P}_{i}\|_{1}<\delta$. The smallest $M$ denoted $M_{\delta}$ is called the covering number of $\mathcal{P}$. Further, $\mathcal{H}(\mathcal{P},\delta)\triangleq\log M_{\delta}$ is called the metric entropy.

Note that in many practical scenarios, the available data at each time $t\in\mathbb{N}$ is delayed, and a data of size $w_{t}$, $t\in\mathbb{N}$ delayed by $D$ slots will be used for estimation/inference purposes [4, 17]. The reason for making the sample size $w_{t}$ depend on $t$ becomes apparent later. Since $p_{k}(t)$, $k=0,1,2,\ldots,K$ depend on $Y_{c}(t)$ for all $t$ (see (2)), we have that the process $p_{k}(t)$ in general is a stochastically dependent sequence. The “degree” of correlation depends on the algorithm used. For $k=0,1,2\ldots,K$ and $s\in\mathbb{N}$, let $\mathbb{P}^{\texttt{ALG},k}_{t,t+s}(*\left|\right.\mathcal{E})$ and $\mathbb{P}^{\texttt{ALG},k}_{t}(*\left|\right.\mathcal{E})$ denote the joint and marginal distributions of $(p_{k}(t),p_{k}(t+s))$ and $p_{k}(t)$ conditioned on the event $\mathcal{E}$, respectively, induced by any algorithm ALG.111In this paper, we propose a distributed Approximate DPP (ADPP) algorithm, and hence ALG will be ADPP. Note that if $p_{k}(t)$ and $p_{k}(t+s)$ are independent for each $t\in\mathbb{N}$ conditioned on some event $\mathcal{E}$, then $\left|\left|\mathbb{P}^{\texttt{ALG},k}_{t,t+s}(*\left|\right.\mathcal{E})-\mathbb{P}^{\texttt{ALG},k}_{t}(*\left|\right.\mathcal{E}){\otimes}\mathbb{P}^{\texttt{ALG},k}_{t+s}(*\left|\right.\mathcal{E})\right|\right|_{\texttt{TV}}=0$. Thus, the difference above, maximized over all slots $t\in\mathbb{N}$ is a natural way of measuring the correlation between the sequences that are $s$ time slots away. More precisely, we have the following definition (see [21] for a related definition).

Definition 2: The $\beta_{1}$ mixing coefficient of the process $p_{k}(t)$, $k=0,1,2,\ldots,K$ conditioned on some event $\mathcal{E}$ is given by

[TABLE]

where $\mathbb{M}_{t,s,k}(\mathcal{E})\triangleq\mathbb{P}^{\texttt{ALG},k}_{t,t+s}(*\left|\right.\mathcal{E})-\mathbb{P}^{\texttt{ALG},k}_{t}(*\left|\right.\mathcal{E})\otimes\mathbb{P}^{\texttt{ALG},k}_{t+s}(*\left|\right.\mathcal{E})$, $s\geq 0$, $\alpha\geq 0$, $\mathbb{P}^{\texttt{ALG},k}_{t}\otimes\mathbb{P}^{\texttt{ALG},k}_{t+s}$ denotes the product distribution, and $\|*\|_{\texttt{TV}}$ is the total variational norm.

Note that in the definition of $\beta_{\texttt{ALG},k}(s,\alpha\left|\right.\mathcal{E})$, we have used $t\geq\alpha$, which is required later in the proof of our main results. Further, if $s$ is large, and the process is sufficiently mixing, then we expect that $\beta_{\texttt{ALG},k}(s,\alpha\left|\right.\mathcal{E})=0$. This definition will be used to decouple a dependent stochastic process so that some of the large deviation bounds that are valid for independent sequences can be applied. The details of this approach will be clear in the proof of our main results. For notational convenience, let us denote the maximum and minimum values of $p_{k}(t)$, $k=0,1,2,\ldots,K$ by $p_{\text{max},k}$ and $p_{\text{min},k}$, respectively. Further, let $(\Delta p)_{\text{max},k}\triangleq p_{\text{max},k}-p_{\text{min},k}$. In the following section, we propose an Approximate DPP (ADPP) algorithm with the associated theoretical guarantees. The $\beta_{1}$ coefficient for the ADPP algorithm will be $\beta_{\texttt{ADPP},k}(s,\alpha\left|\right.\mathcal{E})$.

III Algorithm and Main Results

In the following subsection, we prove that the optimal solution to $\mathbf{P_{0}}$ is close to a LP.

III-A Approximately Optimal LP

Since the number of possible values that $p_{k}(t)$, $k=0,1,2,\ldots,K$ take is finite, the number of possible strategies is also finite.222Due to this, the question of whether $p_{k}(t)$ is convex or not does not matter. The approximate algorithm that we are going to propose chooses one of the pure strategy $\mathbf{S(\omega)}\triangleq\{\mathbf{s}_{1}(\omega_{1}),\mathbf{s}_{2}(\omega_{2}),\ldots,\mathbf{s}_{N}(\omega_{N})\}$ based on the common information $Y_{c}(t)$, where $\mathbf{s}_{i}(\omega_{i})\in\mathcal{A}_{i}$, and $\omega_{i}\in\Omega_{i}$, $i=1,2,\ldots,N$. For example, $\mathbf{s}_{i}(\omega_{i})$ can be a simple threshold rule with the thresholds coming from a finite set. The control action $\alpha_{i}(t)$ at the user $i$ is chosen as a deterministic function of $\omega(t)$, i.e., $\alpha_{i}(t)\triangleq\mathbf{s}_{i}(\omega_{i}(t))$ for all $i\in\{1,2,\ldots,N\}$ and for all $t\in\mathbb{N}$. Let the total number of such pure strategies be $F\triangleq\prod_{i=1}^{N}\left|\mathcal{A}_{i}\right|^{\left|\Omega_{i}\right|}$. Enumerating the $F$ strategies, we get $\mathbf{S}^{m}(\omega)$, $m\in\{1,2,\ldots,F\}$ and $\omega\in\Omega$. Each $\omega\in\Omega$ and the strategy $\mathbf{S}^{m}(\omega)$ result in a cost $p_{k}(\mathbf{S}^{m}(\omega),\omega)$, $k=0,1,2,\ldots,K$. Note that it is possible to reduce $F$ if the problem has a specific structure [4]. For each strategy $m\in\{1,2,\ldots,F\}$, define the average cost/penalty as

[TABLE]

where $k=0,1,2,\ldots,K$ and the underlying distribution of $\omega\in\Omega$ is $\pi^{{}^{\prime}}\in\mathcal{P}_{c}$. As in [4], we consider a randomized algorithm where the strategy $m\in\{1,2,\ldots,F\}$ is picked with probability $\theta_{m}(t)$ in an independent fashion across time $t$. Here, $\theta_{m}(t)$ is a function of the common information $Y_{c}(t)$. The corresponding average cost/penalty at time $t$ becomes

[TABLE]

where $\lambda\in\{\pi_{t},\pi,\mathcal{P}_{i}\}$, $i=1,2,\ldots,M_{\delta}$. In [4], it was shown that the problem $\mathbf{P_{0}}$ when $\pi_{t}=\pi$ for all $t\in\mathbb{N}$ ($\omega(t)$ is i.i.d.) is equivalent to the following LP:

[TABLE]

In this paper, from Assumption 1, we have $\|\pi_{t}-\pi\|_{1}\rightarrow 0$, as $t\rightarrow\infty$. With dense covering of the space $\mathcal{P}$, we expect that the limiting distribution is well approximated by $\mathcal{P}_{i}$ for some $i=1,2,\ldots,M_{\delta}$ in the covering set. More preciesely,

[TABLE]

and the corresponding distance be $d_{\pi,\mathcal{P}_{i^{*}}}\triangleq\|\pi-\mathcal{P}_{i^{*}}\|_{1}<\delta$. Since the distribution of $\omega(t)$ is changing across time, directly applying Theorem $1$ of [4] is not possible. However, from Assumption $1$, we know that the distribution approaches a fixed measure $\pi\in\mathcal{P}_{c}$. Hence, we expect that the algorithm designed for $\pi\in\mathcal{P}_{c}$ or an approximation of $\pi$, i.e., $\mathcal{P}_{i^{*}}$ should eventually be close to the optimal algorithm. Therefore, we consider the following LP denoted $\mathbf{LP_{\mathcal{P}_{i^{*}}}}$:

[TABLE]

Also, we assume that the solution to $\mathbf{LP_{\mathcal{P}_{i^{*}}}}$ exists and the optimal cost is absolutely bounded. Further, define

[TABLE]

where $\Theta\triangleq(\theta_{1},\theta_{2},\ldots,\theta_{F})$, and for any $x\geq 0$,

[TABLE]

where $\mathbf{1}\triangleq\{1,1,\ldots,1\}\in\mathbb{R}^{F}$. Note that $G(0)$ corresponds to $\mathbf{LP_{\mathcal{P}_{i^{*}}}}$. We make the following important smoothness assumption about the function $G(x)$.

Assumption 2: The function $G(x)$ is $c$-Lipschitz continuous around the origin, i.e., for some $c>0$, we have

[TABLE]

In the theorem to follow, given that Assumption 2 is valid, we prove that the optimal cost of the linear optimization problem in (6) is “close” to the optimal cost of $\mathbf{P_{0}}$.

Theorem 1

Let $p^{\text{(opt)}}$ and $p^{(\text{opt})}_{\mathcal{P}_{i^{*}}}$ be the optimal solution to the problems $\mathbf{P_{0}}$ and $\mathbf{{LP}_{\mathcal{P}_{i^{*}}}}$, respectively. Then, under Assumption 2, we have $p^{(\text{opt})}_{\mathcal{P}_{i^{*}}}<p^{\text{(opt)}}+(c+1)\Delta_{\pi,\mathcal{P}_{i^{*}}}$, where for any $\nu>0$, $\Delta_{\pi,\mathcal{P}_{i^{*}}}\triangleq\max_{k=0,1,2,\ldots,K}b_{\text{max},k}(d_{\pi,\mathcal{P}_{i^{*}}}+\nu),\text{ and }b_{\text{max},k}\triangleq\max\{\left|p_{\text{max},k}\right|,\left|p_{\text{min},k}\right|\}.$

Proof: See Appendix B. $\blacksquare$

III-B Approximate DPP (ADPP) Algorithm

In this subsection, we present an online distributed algorithm that approximately solves the problem $\mathbf{P_{0}}$. We assume that at time $t\in\mathbb{N}$, all nodes receive feedback specifying the values of all the penalties and the states, namely, $p_{1}(t-D),p_{2}(t-D),\ldots,p_{K}(t-D)$ and $\omega(t-D)$. Recall that $D\geq 0$ is the delay in the feedback. Using this information, we construct the following set of queues

[TABLE]

$k=1,2,\ldots,K$, and $t\in\mathbb{N}$. These queues act as the common information, i.e., $Y_{c}(t)=\mathbf{Q}_{t}$, where $\mathbf{Q}_{t}\triangleq(Q_{1}(t),Q_{2}(t),\ldots,Q_{K}(t))$. Further, the past $w_{t}$ samples of $\omega(t)$ given by $\{\omega(t-i)$, $i=D,D+1,\ldots,D+w_{t}-1\}$ will be used to find an estimate of the state probabilities which is required for the algorithm that we propose. For all $k=1,2,\ldots,K$, we let $p_{k}(t)=0$ when $t\in\{-1,-2,\ldots,-D\}$. The Lyapunov function is defined as

[TABLE]

and the corresponding drift is given by $\Delta(t)\triangleq\mathcal{L}(t+1)-\mathcal{L}(t)$ for all $t\in\mathbb{N}$. A higher value of the drift indicates that the constraints have been violated frequently in the past. Thus, the control action should be taken that simultaneously minimizes the drift and the penalty (cost). The DPP algorithm tries to find the optimal control action that minimizes an upper bound on the DPP term $\mathbb{E}\left[\Delta(t+D)+Vp_{0}(t)\left|\right.\mathbf{Q}_{t}\right]$, $V\geq 0$, which is the essence of the following lemma. The proof of the lemma follows directly from the proof of Lemma $5$ of [4], and hence omitted.

Lemma 1

For a fixed constant $V\geq 0$, we have

[TABLE]

where $\mathcal{C}_{i,k,t}\triangleq\sum_{m=1}^{F}\beta_{m}(t)r_{k,\pi_{t}}^{(m)}-c_{k}$, $r_{k,\pi_{t}}^{(m)}\triangleq\sum_{\omega\in\Omega}\pi_{t}(\omega)p_{k}({\mathbf{S}^{m}}(\omega),\omega)$, $k=0,1,2,\ldots,K$,

[TABLE]

and, with a slight abuse of notation, $\beta_{m}(t)$ is the probability with which the strategy $m$ is used at time $t$.

Note that as $t\rightarrow\infty$, $B_{t}\rightarrow B$. The expression for $B$ can be obtained by replacing $\pi_{t}(\omega)$ by $\pi(\omega)$ in the expression for $B_{t}$. The algorithm to follow requires an estimate of $\pi_{t}(\omega)$, which can be computed using the past $w_{t}$ samples by means of any estimate such as the sample average. However, when the space $\mathcal{P}$ is “simple”, one can expect to compute an estimate of $\pi_{t}(\omega)$ more efficiently. For example, if the nature chooses $\omega(t)$ from a finite set of distributions ($M_{\delta}<\infty$ for all $\delta>0$), then estimating the distribution corresponds to a hypothesis testing problem. Hence, by approximating the measure space $\mathcal{P}$ by a finite set of measures $\mathcal{P}_{c}$ gives us the flexibility to run a hypothesis testing to find an approximate distribution based on the available $w_{t}$ samples through a likelihood ratio test. In the following, we provide the algorithm.

• •

Algorithm: Given the delayed feedback of size $w_{t}$ at time slot $t\in\mathbb{N}$, i.e., $\omega(t-i-D)$, and $p_{k}(t-D)$, $i=0,1,\ldots,w_{t}-1$ and for $k=1,2,\ldots,K$, perform the following steps

• –

Step 1: Find the probability measure in $\mathcal{P}_{c}$ that best fits the data, i.e., pick $\mathcal{P}_{j_{t}^{*}}\in\mathcal{P}_{c}$ such that

[TABLE]

• –

Step 2: Choose $m_{t}\in\{1,2,\ldots,F\}$ (breaking ties arbitrarily) that minimizes the following:

[TABLE]

• –

Step 3: Set $t\rightarrow t+1$, receive the delayed feedback, update the queues using (9), and go to Step 1.

We say that there is an error in the outcome of step $1$ of the algorithm if $\mathcal{P}_{j_{t}^{*}}\neq\mathcal{P}_{i^{*}}$. Recall that $i^{*}$ corresponds to the index of the probability measure in the covering set that is close to $\pi$ in the $\mathcal{L}_{1}$ norm sense. The error event $\mathcal{E}_{\delta,t}$, $t\in\mathbb{N}$ is defined as those outcomes for which $j_{t}^{*}\neq i^{*}$. Further, let $\mathcal{E}_{[\tau:\tau+s]}\triangleq\bigcup_{t=\tau}^{\tau+s}\mathcal{E}_{\delta,t}$ to denote that there is an error in at least one of the time slot in the interval $\tau$ to $\tau+s$. In the following theorem, we state and prove our first result that will be used to prove the PAC type bound for the ADPP algorithm.

Theorem 2

For the ADPP algorithm, for any $\epsilon_{k}>\frac{1}{t}\sum_{\tau=0}^{t-1}\mathbb{E}p_{k}(\tau)-c_{k}+\frac{{\alpha_{t}}{(p_{\text{max},k}-p_{\text{min},k})}}{t-\alpha_{t}}$, and for constants $\alpha_{t}\in\mathbb{N}$, $u_{t}\in\mathbb{N}$ and $v_{t}\in\mathbb{N}$ such that $v_{t}u_{t}=t-\alpha_{t}$, we have

[TABLE]

*where $\bar{\epsilon}_{t,k}\triangleq\frac{t\epsilon_{t,k}-{\alpha_{t}}(p_{\text{max},k}-p_{\text{min},k})}{t-\alpha_{t}}$, $\epsilon_{t,k}\triangleq\epsilon_{k}+c_{k}-\frac{1}{t}\sum_{\tau=0}^{t-1}\mathbb{E}p_{k}(\tau)$. Here, $c_{0}=p^{(opt)}$, and $c_{k}$, $k=1,2,\ldots,K$ are the constraint variables in $\mathbf{P_{0}}$. *

Proof: See Appendix C. $\blacksquare$

The first term in the bound in Theorem 2 corresponds to the large deviation bound when $p_{k}(t)$’s are independent. The second term corresponds to an upper bound on the probability of error in the time slots $\alpha_{t}$ to $t$ for decoding the correct index $i^{*}$; equivalently, this corresponds to an “incorrect” estimate of the distribution of the states in these slots. The last term captures the stochastic dependency of $p_{k}(t)$ across time $t\in\mathbb{N}$. In order to prove a high probability result, we need to find an expression for each of the terms in the bound. Next, we upper bound the error term $\Pr\{\mathcal{E}_{\delta,\tau}\}$ using the following assumption about the probability space $\mathcal{P}_{c}$. This will come handy in the proof of Lemma 2 below.

Assumption 3: Assume that for all $j=1,2,\ldots,M_{\delta}$, $\mathcal{P}_{j}(\omega)\neq 0$, there exist constants $\alpha_{\delta}>\beta_{\delta}>0$, such that $\alpha_{\delta}>\mathcal{P}_{j}(\omega)>\beta_{\delta}>0$ for all $\omega\in\Omega$.

We use the above assumption in the proof of the following lemma to bound the probability of error term in (15).

Lemma 2

An upper bound on the probability of error is given by

[TABLE]

where $q^{(\tau)}_{e,\texttt{up}}\triangleq\exp\left\{-{2\zeta_{\delta}\mathcal{D}_{\tau}^{2}w_{\tau}}+\mathcal{H}(\mathcal{P},\delta)\right\}$, $\zeta_{\delta}\triangleq{\left[\log\left(\frac{\alpha_{\delta}}{\beta_{\delta}}\right)\right]^{2}}$,

[TABLE]

$\mathcal{D}_{\tau}\triangleq\min_{j\neq i^{*}}\mathcal{D}_{\tau,j}$, and $\mathcal{H}(\mathcal{P},\delta)=\log M_{\delta}$ is the metric entropy. Further, when $u_{t}=\mathcal{O}(\sqrt{{t}})$, $v_{t}=\mathcal{O}(\sqrt{{t}})$, and $\alpha_{t}=\mathcal{O}(\sqrt{{t}})$, we have $\sum_{\tau=\alpha_{t}}^{t-1}\Pr\{\mathcal{E}_{\delta,\tau}\}\preceq(t-\alpha_{t})S_{t,\delta}$, where $S_{t,\delta}\triangleq\exp\left\{-{\phi_{\tau,t,\delta}}+\mathcal{H}(\mathcal{P},\delta)\right\}$. In the above, $\phi_{\tau,t,\delta}\triangleq 2\zeta_{\delta}\left[\min_{\alpha_{t}\leq\tau\leq t}\mathcal{D}_{\tau}\right]^{2}N_{[\alpha_{t}:t]}$, $N_{[\alpha_{t}:t]}\triangleq\min_{\alpha_{t}\leq\tau\leq t}w_{\tau}$.

Proof: See Appendix D. $\blacksquare$

From the above lemma, we have that the error goes to zero exponentially fast as $\tau\rightarrow\infty$. The fact that $\sum_{\tau=\alpha_{t}}^{t-1}\Pr\{\mathcal{E}_{\delta,\tau}\}\preceq(t-\alpha_{t})S_{t,\delta}\rightarrow 0$ exponentially fast as $t\rightarrow\infty$ will be used later in the paper to prove the almost sure convergence of the algorithm to the optimal. Now, it remains to find an upper bound on the first and the last term in (15). The following theorem uses the Assumption 3, (15) and (16) to provide a PAC result for the above algorithm in terms of the $\beta_{1}$ coefficient.

Theorem 3

Under Assumptions 1-3, for the proposed Algorithm with $\epsilon_{0}=(c+1)\Delta_{\pi,\mathcal{P}_{i^{*}}}+\psi_{t}(\delta)+\frac{{\alpha_{t}}{(p_{\text{max},k}-p_{\text{min},k})}}{t-\alpha_{t}}+\epsilon$, $\epsilon_{k}=Q_{\texttt{up}}(t)+\epsilon$, $k=1,2,\ldots,K$, and some finite positive constants $V$, $C$ and $c$, the following holds.

For every $\epsilon>0$, with a probability of at least $1-\gamma_{0}$,

[TABLE]

provided $t\in\mathcal{T}_{t,0}$. Here, $\gamma_{0}>\beta_{0}^{*}$, $\beta_{0}^{*}\triangleq(t-\alpha_{t})\left[\beta_{\texttt{ADPP},0}(u_{t},\alpha_{t}\left|\right.\mathcal{E}_{[\alpha_{t}:t]})+S_{t,\delta}\right]$, where $S_{t,\delta}$ is as defined in earlier. 2. 2.

For every $\epsilon>0$, with a probability of at least $1-\gamma_{1}$,

[TABLE]

$k=1,2,\ldots,K$*, *

provided $t\in\mathcal{T}_{t,1}$. Here $\gamma_{1}>\beta_{1}^{*}$, where

[TABLE]

In the above,

[TABLE]

$i\in\{0,1\}$, $\Delta_{\pi,\mathcal{P}_{i^{*}}}=\max_{k=0,1,2,\ldots,K}b_{\text{max},k}(d_{\pi,\mathcal{P}_{i^{*}}}+\nu)$, and

[TABLE]

where $\rho\triangleq\sum_{k=1}^{K}(p_{\text{max},k}-c_{k})^{2}$, $\bar{J}_{t}\triangleq\max_{0\leq k\leq K}p_{\text{max},k}\left(\frac{1}{t}\sum_{\tau=0}^{t-1}\|\pi_{\tau}-\pi\|_{1}+\delta\right)$, $\bar{H}_{t}\triangleq\frac{1+2D}{t}\sum_{\tau=0}^{t-1}B_{\tau}$. Further, $\mathcal{D}_{\tau,j}$, $\mathcal{D}_{\tau}$, $\zeta_{\delta}$, and $P_{e,\texttt{up}}^{(\tau)}$ are as defined in Lemma 2. Also, $Q_{\texttt{up}}(t)\triangleq\sqrt{\frac{VF}{t}+\frac{\Gamma_{t}}{t^{2}}}$ and $\Gamma_{t}\triangleq V(c+1)(\Delta_{\pi,\mathcal{P}_{i^{*}}}+{{\bar{J}_{t}})+\bar{H}_{t}+C}+(1+2D)\sum_{\tau=0}^{t-1}B_{\tau}P_{e,\texttt{up}}^{(\tau)}+p_{\text{max},0}\sum_{\tau=0}^{t-1}P_{e,\texttt{up}}^{(\tau)}+{\rho}\sum_{\tau=0}^{t-1}\tau P_{e,\texttt{up}}^{(\tau)}$ and $p_{\text{max,k}}$, $k=1,2,\ldots,K$ is as defined earlier.

Proof: See the Appendix E. $\blacksquare$

The above result can be used to provide almost sure convergence as well as finite sample complexity result provided we show that the $\beta_{1}$ mixing coefficient decays sufficiently fast. This requires us to prove a bound on $\beta_{\texttt{ADPP},k}$. First, we consider a special case of the centralized scheme, i.e., $D=0$. Then, we extend the proof to any $D\geq 0$. The details of this are provided next.

IV Bound On the Mixing Coefficient

By using the Pinsker’s inequality that relates the total variational norm and the mutual information, we have the following bound [22]

[TABLE]

where $X_{k,t}\triangleq p_{k}(t)$, $I(X_{k,t};X_{k,t-s}\left|\right.\mathcal{E}_{[\alpha_{t}:t]}^{c})$ is the mutual information between random variables $p_{k}(t)$ and $p_{k}(t-s)$, $k=0,1,2,\ldots,K$ conditioned on $\mathcal{E}_{[\alpha_{t}:t]}^{c}$, and any $s\in\mathbb{N}$. Later, we use $s=u_{t}$, as required. Thus, proving an upper bound on $\beta_{\texttt{ADPP},k}(s,\alpha_{t}\left|\right.\mathcal{E}_{[\alpha_{t}:t]}^{c})$ amounts to finding an upper bound on the conditional mutual information. To present our results, we use the following notations. Let $\mathbf{X}_{t}\triangleq(X_{0,t},X_{1,t},\ldots,X_{K,t})$, $\mathbf{X}_{\neq k,t}\triangleq(X_{1,t},X_{2,t}\ldots,X_{k-1,t},X_{k+1,t},\ldots,X_{K,t})$, and as before, $\mathbf{Q}_{t}\triangleq(Q_{1}(t),Q_{2}(t),\ldots,Q_{K}(t))$. We first note that

[TABLE]

where the last inequality follows from the fact that the mutual information is non-negative. Thus, we have

[TABLE]

Let $\mathcal{Q}_{t}$ be the set of all vectors that $\mathbf{Q}_{t}$ takes at time $t$. Also, let $\mathcal{M}_{t}:\mathcal{Q}_{t}\rightarrow\{1,2,\ldots,F\}$ be the rule induced by the ADPP algorithm that determines the strategy given the queue at time $t$. In order to obtain an upper bound on the mutual information, we state the following assumption about the conditional distribution of the process $\omega(t)$.

Assumption 4: For some $\kappa>0$, $\mathbf{Q}_{t}\in\mathcal{Q}_{t}$, and for all $t\in\mathbb{N}$, we assume that the following bound is satisfied

[TABLE]

Note that a lower value of $\kappa$ signifies the fact that the channel is noisy. For example, when $\kappa=0$, we have uniform conditional distribution for all $m$ and $x$ leading to a completely noisy channel from $Q_{\tau}$ to $X_{\tau}$. Next, we present an upper bound on $\beta_{\texttt{ADPP},k}(s,\alpha_{t}\left|\right.\mathcal{E}_{[\alpha_{t}:t]}^{c})$ for the $D=0$ case (centralized scheme).

IV-A Bound on $\beta_{\texttt{ADPP},k}(s,\alpha_{t}\left|\right.\mathcal{E}_{[\alpha_{t}:t]}^{c})$ when $D=0$

In order to get insights on the proof of bounding the $\beta_{1}$ coefficient for the general scenario of $D\geq 0$, we first consider the centralized scheme, i.e., $D=0$, and later we provide proofs and results for the $D>0$ case. For $D=0$, the queue update in the vector form becomes

[TABLE]

where $\mathbf{C}\triangleq(c_{1},c_{2},\ldots,c_{K})$. Recall that Step 2 of the Algorithm uses $\mathbf{Q}_{t}$ and the output from Step 1 to find a pure strategy in a deterministic fashion that maximizes an upper bound on the drift-plus-penalty expression. Thus, the strategy is a deterministic function of the queue. Note that conditioned on the event $\mathcal{E}_{[\alpha_{t}:t]}^{c}$, the output of Step 1 is $i^{*}$ for all time slots $\tau\in\{\alpha_{t},\ldots,t\}$. Conditioned on $\mathcal{E}_{[\alpha_{t}:t]}^{c}$, this leads to the following Markov chain model

[TABLE]

Fig. 2 depicts the graphical model representation of the above. In order to prove an upper bound on the mutual information, we use the Strong Data Processing Inequality (SDPI) for the graphical model shown in Fig. 2.

Note that the Assumption 4 facilitates the analysis of the $\beta_{1}$ mixing coefficient, and is also related to the differential privacy constraint in [23]. The following theorem provides an upper bound on the mixing coefficient.

Theorem 4

Given Assumption 4, for $D=0$, $\kappa<\log 3$, and for any $t\geq s\geq\alpha_{t}$, an upper bound on the $\beta_{1}$ mixing coefficient is given by

[TABLE]

where $\mu\triangleq F\left|\Omega\right|(K+1)$ is the number of possible values that $\mathbf{X}_{t}$ can take, $t\in\mathbb{N}$, and $\theta\triangleq\max\left\{\frac{(e^{\kappa}-1)}{2},\frac{1}{2}\right\}<1$.

Proof: See the Appendix F. $\blacksquare$

Note that $s=u_{t}$, and suppose $u_{t}$ grows with $t$, then the Theorem says that the mixing coefficient goes down to zero exponentially fast with $t$. Thus, we have the following important corollary.

Corollary 1

Given Assumption 4, for $D=0$, $u_{t}=\mathcal{O}(\sqrt{t})$, $\kappa<\log 3$, and for any $t\geq s\geq\alpha_{t}$, an upper bound on the $\beta_{1}$ mixing coefficient is given by

[TABLE]

where $\mu\triangleq F\left|\Omega\right|(K+1)$ is the number of possible values that $\mathbf{X}_{t}$ can take, $t\in\mathbb{N}$, and $\theta\triangleq\max\left\{\frac{(e^{\kappa}-1)}{2},\frac{1}{2}\right\}<1$.

A finite time bound can easily be obtained by substituting the upper bound of Theorem 4 in Theorem 3. However, in order to get more insights into the main result of this paper, we will look at the asymptotic in the following subsection.

IV-A1 Asymptotics

Note that when $D=0$, the authors in [19] prove the convergence of the algorithm to the optimal in probability. Here, we use a different approach compared to [19] to show an almost sure as well as a high probability convergence of the the proposed ADPP algorithm to the optimal, when $D=0$. By using the insights obtained here, we generalize the result to an arbitrary $D\geq 0$ in the subsequent subsections. First, in the following lemma, we provide a high probability guarantees of the ADPP algorithm when $D=0$ and $t\rightarrow\infty$.

Lemma 3

Under Assumptions 1-4, for the proposed Algorithm with $D=0$, $\alpha_{t}=\mathcal{O}(\sqrt{t})$, $w_{t}=\mathcal{O}(\sqrt{t})$, $V=\mathcal{O}(\sqrt{t})$, $\kappa<\log 3$, and some finite positive constant $c$, the following holds.

• •

For every $\epsilon>0$, we have

[TABLE]

*and ** $\lim_{t\rightarrow\infty}\Pr\left\{\frac{1}{t}\sum_{\tau=0}^{t-1}p_{k}(\tau)\leq c_{k}+\epsilon\right\}=1$, k = 1,2,…,K. *

In the above, $\Delta_{\pi,\mathcal{P}_{i^{*}}}=\max_{k=0,1,2,\ldots,K}b_{\text{max},k}(d_{\pi,\mathcal{P}_{i^{*}}}+\nu)$,

[TABLE]

and $p_{\text{max,k}}$, $k=1,2,\ldots,K$ is as defined earlier.

Proof: See Appendix G. $\blacksquare$

The interpretations of the above result will be provided later. Next, we use Lemma 2 along with the Borel-Cantelli Lemma to provide an almost sure convergence of the ADPP algorithm.

Theorem 5

Under Assumptions 1-4, for the proposed Algorithm with $D=0$, $\alpha_{t}=\mathcal{O}(\sqrt{t})$, $w_{t}=\mathcal{O}(\sqrt{t})$, $V=\mathcal{O}(\sqrt{t})$, $\kappa<\log 3$, and some finite positive constant $c$, the following holds.

• •

For every $\epsilon>0$, almost surely, we have

[TABLE]

[TABLE]

In the above, $\Delta_{\pi,\mathcal{P}_{i^{*}}}$, $\bar{J}$, and $p_{\text{max,k}}$, $k=1,2,\ldots,K$ are as defined earlier.

Proof: See Appendix H. $\blacksquare$

From Theorems 3 and 5, it is easy to see that the error can be reduced by reducing $\Delta_{\pi,\mathcal{P}_{i^{*}}}$, which amounts to reducing $d_{\pi,\mathcal{P}_{i^{*}}}$ and $\nu$. Note that $d_{\pi,\mathcal{P}_{i^{*}}}<\delta$ can be reduced by reducing the error in the covering of the probability space $\mathcal{P}_{c}$. This comes at a cost of increased metric entropy since $\delta$ needs to be reduced. However, as $t\rightarrow\infty$, increased metric entropy does not effect the overall result. Further, a lower value of $\bar{J}$ signifies lesser error. This is possible only when the rate at which the probability measure $\pi_{t}$ converges to $\pi$ is “sufficiently” high. In particular, this is true when $\sum_{\tau=0}^{t-1}\|\pi_{\tau}-\pi\|_{1}=\mathcal{O}(t^{\zeta})$, where $\zeta<1$. In the next subsection, we provide an almost sure as well as high probability result for any $D\geq 0$.

IV-B Bound on $\beta_{\texttt{ADPP},k}(s,\alpha_{t}\left|\right.\mathcal{E}_{[\alpha_{t}:t]}^{c})$ when $D\geq 0$

As in the previous subsection, we use $s=u_{t}$. For $D\geq 0$, the queue update in the vector form is given by

[TABLE]

where $\mathbf{C}\triangleq(c_{1},c_{2},\ldots,c_{K})$ and $\mathbf{X}_{\neq 0,t-D}$ is as defined earlier in this section. As in the $D=0$ case, we condition on the event $\mathcal{E}_{[\alpha_{t}:t]}^{c}$, and therefore, the output of Step 1 is $i^{*}$ for all time slots $\tau\in\{\alpha_{t},\ldots,t\}$.333Recall that $i^{*}$ is the index corresponding to $\pi_{i^{*}}$, which is the distribution “close” to $\pi$. Define the following shorthand notations $\mathbf{Q}_{1:n}\triangleq\{\mathbf{Q}_{1},\mathbf{Q}_{2},\ldots,\mathbf{Q}_{n}\}$, and $\mathbf{X}_{1:m}\triangleq\{\mathbf{X}_{1},\mathbf{X}_{2},\ldots\mathbf{X}_{m}\}$. Unlike the $D=0$ case, conditioning on $\mathcal{E}_{[\alpha_{t}:t]}^{c}$ leads to the following Markov chain model

[TABLE]

where $l_{s}\triangleq\lceil\frac{s+1}{D}\rceil$. Fig. 3 depicts the graphical model representation of the above.

Note that the $i^{th}$ pair in the Markov chain is $(\mathbf{Q}_{t-(i+1)D+1:t-iD},X_{t-(i+1)D+1:t-iD})$, $i=0,1,\ldots,l_{s}-1$. In order to prove an upper bound on the mutual information, we use the Strong Data Processing Inequality (SDPI) for the graphical model shown in Fig. 3. Using the above mentioned Markov property, we need to bound the term $I(\mathbf{X}_{t};\mathbf{X}_{t-s}\left|\right.\mathcal{E}_{[\alpha_{t}:t]}^{c})$ from (22). In the following theorem, we present a bound on the $\beta_{1}$ mixing coefficient for $D\geq 0$ case.

Theorem 6

Given Assumption 4, for $D\geq 0$, $D<\frac{\log 3}{\kappa}$, and for any $t\geq s\geq\max\{\alpha_{t},2D+1\}$, an upper bound on the $\beta_{1}$ mixing coefficient is given by

[TABLE]

$k=0,1,2,\ldots,K$, where $\mu_{D}\triangleq DF\left|\Omega\right|(K+1)$ is the number of possible values that $X_{t}$ can take, $t\in\mathbb{N}$, and $\theta\triangleq\max\left\{\frac{(e^{{\kappa}D}-1)}{2},\frac{1}{2}\right\}<1$.

Proof: For the ease of notation, let $\mathbf{X}_{i,t}\triangleq\mathbf{X}_{t-(i+1)D+1:t-iD}$ and $\mathbf{Q}_{i,t}\triangleq\mathbf{Q}_{t-(i+1)D+1:t-iD}$. First, in the following, we prove that $I(\mathbf{X}_{t};\mathbf{X}_{t-s}\left|\right.\mathcal{E}_{[\alpha_{t}:t]}^{c})\leq I(\mathbf{X}_{0,t};\mathbf{X}_{l_{s}-1,t}\left|\right.\mathcal{E}_{[\alpha_{t}:t]}^{c})$. Consider

[TABLE]

where $(a)$ and $(b)$ follow from the definitions of $\mathbf{X}_{t}\in\mathbf{X}_{0,t}$ and $\mathbf{X}_{t-s}\in\mathbf{X}_{l_{s}-1,t}$, and the fact that the mutual information is non-negative. We need to upper bound $I(\mathbf{X}_{0,t};\mathbf{X}_{l_{s}-1,t}\left|\right.\mathcal{E}_{[\alpha_{t}:t]}^{c})$, which is obtained in a manner similar to the $D=0$ case, as explained next. Since $\mathbf{X}_{l_{s}-1,t}\rightarrow{\mathbf{Q}_{0,t}}\rightarrow\mathbf{X}_{0,t}$ forms a Markov chain, we obtain the following bound from the SDPI

[TABLE]

where $\eta_{\texttt{ch}_{1}}$ is the Dobrushin’s contraction coefficient for the channel from ${\mathbf{Q}_{0,t}}$ to ${\mathbf{X}_{0,t}}$ defined as

[TABLE]

In the above, $\gamma$ and $\gamma^{{}^{\prime}}$ represent the vector values taken by ${\mathbf{Q}_{0,t}}$. It will be shown later that $\eta_{\texttt{ch}_{1}}<1$. Note that by simple data processing inequality, we have from (33) that

[TABLE]

The first inequality above follows from the fact that ${\mathbf{Q}_{0,t}}$ is a deterministic function of ${\mathbf{X}_{1,t}}$ and ${\mathbf{Q}_{1,t}}$. Since $\mathbf{X}_{l_{s}-1,t}\rightarrow({\mathbf{Q}_{2,t}},{\mathbf{X}_{2,t}})\rightarrow({\mathbf{Q}_{1,t}},{\mathbf{X}_{1,t}})$ forms a Markov chain, the above can be further bounded as follows (see Fig. 3)

[TABLE]

where $\eta_{\texttt{ch}_{2}}$ is the Dobrushin’s coefficient for the channel $({\mathbf{Q}_{2,t}},{\mathbf{X}_{2,t}})$ to $({\mathbf{Q}_{1,t}},{\mathbf{X}_{1,t}})$ defined as

[TABLE]

Note that $\mathbf{X}_{l_{s}-1,t}\rightarrow({\mathbf{Q}_{j,t}},{\mathbf{X}_{j,t}})\rightarrow({\mathbf{Q}_{j-1,t}},{\mathbf{X}_{j-1,t}})$ forms a Markov chain for all $j=2,3,\ldots,l_{s}-2$. The corresponding Dobrushin coefficient is given by

[TABLE]

Using these in (IV-B), and applying the bound repeatedly, we get

[TABLE]

But, $I(\mathbf{X}_{l_{s}-1,t};{\mathbf{Q}_{l_{s}-2,t}},{\mathbf{X}_{l_{s}-2,t}}\left|\right.\mathcal{E}_{[\alpha_{t}:t]}^{c})\leq H(\mathbf{X}_{l_{s}-2,t})\leq\log N_{D}$, where $N_{D}\triangleq DF\left|\Omega\right|(K+1)$ is the number of possible values that $\mathbf{X}_{l_{s}-2,t}$ can take. Using this in the above, we get

[TABLE]

Next, in Lemmas 4 and 5, we prove an upper bound on $\eta_{\texttt{ch}_{j}}$ for every $j=1,2,\ldots,l_{s}-2$.

Lemma 4

Under Assumption $4$, for every $D\geq 0$, and $D<\frac{\log 3}{\kappa}$, we have the following upper bound on $\eta_{\texttt{ch}_{1}}$

[TABLE]

Proof: See Appendix I. $\blacksquare$

Lemma 5

Under Assumption $4$, for every $D\geq 0$, and $D<\frac{\log 3}{\kappa}$, we have the following upper bound

[TABLE]

Proof: See Appendix J. $\blacksquare$

Since there are $l_{s}-1$ terms in the overall product, we need $l_{s}-1\geq 1\Rightarrow s\geq 2D+1$. Using the bounds in Lemmas 4 and 5 in (40), and substituting the result in (22), we get the desired result, which completes the proof of Theorem 6. $\blacksquare$

The above result says that for a given $\kappa$, the ADPP algorithm converges to the optimal provided the delay $D$ in the available samples at each node is bounded by a constant $\frac{\log 3}{\kappa}$.

IV-B1 Almost sure convergence

Using Lemmas 4 and 5, and the main result in Theorem 6, the following result can be obtained in a fashion similar to the $D=0$ case. In the following lemma, we provide a high probability guarantees of the ADPP algorithm for a general $D\geq 1$ and as $t\rightarrow\infty$.

Lemma 6

Under Assumptions 1-3, for the proposed Algorithm with ${D\geq 0}$, $\alpha_{t}=\mathcal{O}(\sqrt{t})$, $w_{t}=\mathcal{O}(\sqrt{t})$, $\kappa<\frac{\log 3}{D}$, and some finite positive constant $c$, the following holds.

• •

For every $\epsilon>0$, we have

[TABLE]

[TABLE]

In the above, $\Delta_{\pi,\mathcal{P}_{i^{*}}}=\max_{k=0,1,2,\ldots,K}b_{\text{max},k}(d_{\pi,\mathcal{P}_{i^{*}}}+\nu)$,

[TABLE]

and $p_{\text{max,k}}$, $k=1,2,\ldots,K$ is as defined earlier.

Proof: The proof is similar to the proof of Lemma 3, and hence omitted. $\blacksquare$

Note that the effect of $D$ is captured through the requirement of $\kappa$, i.e., $\kappa<\frac{\log 3}{D}$. In particular, large values of the delay $D$ require stringent constraint on the “noisyness” of the channel, i.e., a lower value of $\kappa$. However, in many cases, $\kappa$ need not be low, and hence the $\beta_{1}$ coefficient may not converge to zero exponentially. Thus, the ADPP algorithm may not be asymptotically optimal. In the next subsection, we consider the general case of $D>1$. We show that the ADPP algorithm converges to the optimal in the almost sure sense.

Theorem 7

Under Assumptions 1-4, for the proposed Algorithm with $D{\geq}1$, $\alpha_{t}=\mathcal{O}(\sqrt{t})$, $w_{t}=\mathcal{O}(\sqrt{t})$, $\kappa\leq\frac{\log 3}{D}$, and some finite positive constant $c$, the following holds.

• •

For every $\epsilon>0$, almost surely, we have

[TABLE]

[TABLE]

In the above, $\Delta_{\pi,\mathcal{P}_{i^{*}}}$, $\bar{J}$, and $p_{\text{max,k}}$, $k=1,2,\ldots,K$ are as defined earlier.

Proof: The proof is similar to the proof of Theorem 5, and hence omitted. $\blacksquare$

In the next section, we present the simulation results to corroborate some of the observations made in the paper.

V Simulation Results

For the simulation setup, we consider the $3$ sensors example of Sec. II. The problem is to maximize the average of the utility in (1) subject to an average power constraint of $1/3$. Here, the utility is the negative of the cost. The probability measure $\pi_{t}$ is chosen from a set of $8$ distributions, and converges to $\{0.1,0.7,0.1,0.1\}$. Due to lack of space, we skip the details of the distribution that is used in the transient time. The optimal value of this is $p^{(opt)}=0.394$. When $\alpha_{i}(t)=1$, $i=1,2,3$, a power of $1$ watt each is consumed. Figures 4 and 5 show the plots of utility and penalty averaged over $1000$ instantiations, versus time $t$ for different values of $V$, $D=00$ and $w_{t}=40$ for all $t$, demonstrating the tradeoff in terms of $V$. For large values of $t$, the utility achieved by the algorithm with $V=20$ is close to optimum while satisfying the constraints thereby confirming the optimality of the algorithm. It is important to note that the mixing coefficient can be easily estimated, and hence mixing condition can be verified through simulation.

VI Concluding Remarks

In this paper, we considered a distributed stochastic optimization problem with independent and asymptotically stationary states. We showed that this stochastic optimization problem is approximately equal to a LP that is a function of the limiting distribution of the state. For the proposed approximate DPP algorithm, we showed that with certain probabilities $\gamma_{0}$ and $\gamma_{1}$, the average cost and penalties are within constants of the optimal solution and the constraints, respectively, provided the waiting time $t>\text{a threshold}$. The threshold is in terms of the mixing coefficient that indicates the non-stationarity of the cost/penalties. The approximation errors capture the degree of non-stationarity (i.e., $\|\pi_{t}-\pi\|_{1}$), the number of samples used to compute an estimate of the state distribution. Also, we have proved an almost sure convergence of the proposed algorithm to a constant close to the optimal. Finally, we presented simulations results to corroborate our theoretical findings.

Appendix A Mcdiarmid’s Inequality

Theorem 8

(See [24]) Let $Z_{1},Z_{2},\ldots,Z_{n}$ be independent random variables all taking values in the set $\mathcal{Z}$. Let $f:\mathcal{Z}^{n}\rightarrow\mathbb{R}$ be a function that satisfies the following

[TABLE]

for all $i=1,2,\ldots,n$. Then, for all $\epsilon>0$

[TABLE]

Appendix B Proof of Theorem 1

In this proof, we use the fact that by decreasing the objective function and increasing the constraints $c_{k}$, $k=1,2,\ldots,K$ in $\mathbf{P_{0}}$ will result in a decreased optimal value. Consider the cost/penalties of the problem $\mathbf{P_{0}}$

[TABLE]

for $k=0,1,2,\ldots,K$ and any $t^{{}^{\prime}}>0$. In the above, $p_{k}(\tau)\triangleq p_{k}(\alpha(\tau),\omega(\tau))$ and $(a)$ follows by adding and subtracting $\mathcal{P}_{i^{*}}(\omega)$ as mentioned earlier and $\pi(\omega)$. Since $\lim_{t\rightarrow\infty}\|\pi_{t}-\pi\|_{1}=0$, for every $\nu>0$, there exists a $t^{{}^{\prime}}\in\mathbb{N}$ such that for all $t>t^{{}^{\prime}}$, $\|\pi_{t}-\pi\|_{1}<\nu$. Using this $t^{{}^{\prime}}$, and the fact that

[TABLE]

for every $k$ and $t>t^{{}^{\prime}}$, we have

[TABLE]

where $b_{\text{max},k}\triangleq\max\{\left|p_{\text{max},k}\right|,\left|p_{\text{min},k}\right|\}$. Similarly, we have

[TABLE]

where $d_{\pi,\mathcal{P}_{i^{*}}}$ is as defined earlier. Using (51) and (B) in (49), we get the following lower bound for all $k=1,2,\ldots,K$.

[TABLE]

where $\Delta_{\pi,\mathcal{P}_{i^{*}}}=\max_{k=0,1,2,\ldots,K}b_{\text{max},k}(d_{\pi,\mathcal{P}_{i^{*}}}+\nu)$. By using the above lower bound in $\mathbf{P_{0}}$, we get the following optimization problem

[TABLE]

where the expectation is taken with respect to $\mathcal{P}_{i^{*}}$. Note that the optimal cost obtained by solving $\mathbf{P_{1}}$ is smaller than $p^{\text{opt}}$. Further, the term $\Delta_{\pi,\mathcal{P}_{i^{*}}}$ is independent of the control action. It is evident from $\mathbf{P_{1}}$ that it is equivalent to $\mathbf{P_{0}}$ where the states $\omega(t)$ is i.i.d. whose distribution is $\mathcal{P}_{i^{*}}$. Using Theorem $1$ of [4], it is easy to see that the solution to $\mathbf{P_{1}}$ is equal to $G(\Delta_{\pi,\mathcal{P}_{i^{*}}})-\Delta_{\pi,\mathcal{P}_{i^{*}}}$,where $G(x)$ is as defined in (7). Thus, from Assumption 2, we have that

[TABLE]

where $p^{\text{(pert)}}_{\mathcal{P}_{i^{*}}}$ denotes the optimal cost of $\mathbf{P}_{1}$. This leads to $p^{\text{(pert)}}_{\mathcal{P}_{i^{*}}}>p^{(\text{opt})}_{\mathcal{P}_{i^{*}}}-(c+1)\Delta_{\pi,\mathcal{P}_{i^{*}}}$. But, we know that $p^{\text{(pert)}}_{\mathcal{P}_{i^{*}}}\leq p^{\text{(opt)}}$, which implies that $p^{(\text{opt})}_{\mathcal{P}_{i^{*}}}<p^{\text{(opt)}}+(c+1)\Delta_{\pi,\mathcal{P}_{i^{*}}}$. $\blacksquare$

Appendix C Proof of Theorem 2

Fix constants $\alpha_{t}\in\mathbb{N}$, $u_{t}$ and $v_{t}$ such that $u_{t}v_{t}=(t-\alpha_{t})$, as shown in Fig. 1. Let $\bar{p}_{k}(t)\triangleq\frac{1}{t}\sum_{\tau=0}^{t-1}p_{k}(\tau)$. By adding and subtracting $\mathbb{E}\bar{p}_{k}(t)$ in the event in (15), we get

[TABLE]

where $\epsilon_{t,k}\triangleq\epsilon_{k}+c_{k}-\mathbb{E}{\bar{p}_{k}{(t)}}$, and the above inequality follows from the fact that $p_{\text{min},k}\leq p_{k}(t)\leq p_{\text{max},k}$ for all $t\in\mathbb{N}$. Note that we need $\epsilon_{t,k}>\frac{\alpha_{t}}{t}(p_{\text{max},k}-p_{\text{min},k})$, which by using the definition of $\epsilon_{t,k}$ implies that $\epsilon_{k}>\mathbb{E}{\bar{p}_{k}{(t)}}-c_{k}+\frac{\alpha_{t}}{t}(p_{\text{max},k}-p_{\text{min},k})$. In order to apply the well known concentration inequalities, we need $p_{k}(t)$ to be independent across time $t$. Since $p_{k}(t)$’s are dependent across time, we use coupling argument to couple $p_{k}(t)$ process with an independent process $\tilde{p}_{k}(t)$ with the same distribution as $p_{k}(t)$. First, we divide the time slots from $\alpha_{t}$ to $t$ into $v_{t}$ blocks of size $u_{t}$ each. Thus, we need $t-\alpha_{t}=v_{t}u_{t}$. We divide the cost/penalties across blocks as

[TABLE]

$i=0,1,2,\ldots,u_{t}-1$ and $k=0,1,\ldots,K$. See Fig. 1 for an illustration. Now, the time average of cost/penalties in (55) can be written as

[TABLE]

where $\Psi_{k,i,t}\triangleq\frac{1}{v_{t}}\sum_{s\in\mathcal{S}_{i,k}^{(t)}}p_{k}(s)$. Note that each term in $\Psi_{i,k,t}$ is at least $u_{t}$ slots apart. Using this, the above bound can be written as

[TABLE]

where $\bar{\epsilon}_{t,k}\triangleq\frac{t\epsilon_{t,k}-{\alpha_{t}}(p_{\text{max},k}-p_{\text{min},k})}{t-\alpha_{t}}$, $\Delta\Psi_{k,i,t}\triangleq\Psi_{k,i,t}-\mathbb{E}{\Psi_{k,i,t}}$, $\tilde{\Psi}_{k,i,t}\triangleq\frac{1}{v_{t}}\sum_{\tau\in\mathcal{S}_{i,k}^{(t)}}\tilde{p}_{k}(\tau)$, and $\tilde{p}_{k}(\tau)$, conditioned on $\mathcal{E}_{[\alpha_{t}:t]}^{c}$, is an independent stochastic process having the same distribution as $p_{k}(\tau)$, $k=0,1,2,\ldots,K$. In the above, $(a)$ follows from the fact that the convex combination of the terms being greater than a constant implies that at least one of the term should be greater than the constant, and using the union bound. The inequality $(b)$ is obtained by applying proposition $1$ of [21] to the indicator function $g\triangleq\mathbf{1}\{\Delta\Psi_{k,i,t}>\epsilon_{t,k}\}$ with expectation replaced by the conditional expectation $\mathbb{E}\left\{*\left|\right.\mathcal{E}_{[\alpha_{t}:t]}^{c}\right\}$. Recall that the event $\mathcal{E}_{\delta,\tau}$ corresponds to the error in decoding the correct index $i^{*}$ at time $\tau$ in Step 1 of the algorithm, and $\mathcal{E}_{[\alpha_{t}:t]}=\bigcup_{\tau=\alpha_{t}}^{t}\mathcal{E}_{\delta,\tau}$. Note that $\Delta\tilde{\Psi}_{k,i,t}$ is the sum of $v_{t}$ independent random variables. Thus, by applying the Mcdiarmids inequality from Theorem 8 of Appendix A along with the fact that $\tilde{\Psi}_{k,i,t}\leq(\Delta p)_{\max,k}=p_{\text{max},k}-p_{\text{min},k}$ for all $t\in\mathbb{N}$, we get

[TABLE]

The above implies that

[TABLE]

Using this in (57), we get the desired result. $\blacksquare$

Appendix D Proof of Lemma 2

First, we assume that at time $\tau$, $w_{\tau}$ samples are available. This requires $\tau-D-w_{\tau}+1>0$, which implies that $\tau>D+w_{\tau}-1$. Otherwise, the decoder will pick one of the $M_{\delta}$ measures uniformly at random resulting in a probability of error of $1/M_{\delta}$. This results in the second inequality in the theorem. Let $j_{\tau}$ be the output of the Algorithm at time slot $\tau$. In the following, we compute an upper bound on the probability of error when $\tau>D+w_{\tau}-1$, i.e.,

[TABLE]

where $\mathcal{E}_{\delta,\tau}$ is the error in slot $\tau\in\mathbb{N}$ of step $1$ of the ADPP Algorithm due to incorrectly detecting the “right distribution,” $\mathcal{P}_{i^{*}}\in\mathcal{P}_{c}$ (see (13)). In the above, $g_{\tau,D,w_{\tau},j_{\tau}}\triangleq\frac{1}{w_{\tau}}\sum_{s=\tau-D-w_{\tau}+1}^{\tau-D}\log\left(\frac{\mathcal{P}_{j_{\tau}}(\omega(s))}{\mathcal{P}_{i^{*}}(\omega(s))}\right)$, $f_{\tau,D,w_{\tau}}\triangleq\frac{1}{w_{\tau}}\sum_{s=\tau-D-w_{\tau}+1}^{\tau-D}\log\mathcal{P}_{i^{*}}(\omega(s))$, and $\mathcal{D}_{\tau,j_{\tau}}$ is as defined in the Lemma. Note that $(a)$ follows from the union bound. By using the following boundedness property from Assumption 3, i.e., $\log\left(\frac{\mathcal{P}_{j_{\tau}}(\omega(s))}{\mathcal{P}_{i^{*}}(\omega(s))}\right)\leq\log\left(\frac{\alpha_{\delta}}{\beta_{\delta}}\right)$, and using the Hoeffdings inequality (see [24]), we get

[TABLE]

where $\zeta_{\delta}\triangleq{\left[\log\left(\frac{\alpha_{\delta}}{\beta_{\delta}}\right)\right]^{2}}$, $\mathcal{D}_{\tau}\triangleq\min_{j_{\tau}\neq i^{*}}\mathcal{D}_{\tau,j_{\tau}}$, and $\mathcal{H}(\mathcal{P},\delta)=\log M_{\delta}$ is the metric entropy. Assuming $\alpha_{t}=\mathcal{O}(\sqrt{t})$ and $w_{t}=\mathcal{O}(\sqrt{t})$, we have that $\alpha_{t}>D+w_{\alpha_{t}}-1=D+t^{1/4}-1$. This implies that $\Pr\{\mathcal{E}_{\delta,\tau}\}\preceq\exp\left\{-{2\zeta_{\delta}\mathcal{D}_{\tau}^{2}w_{\tau}}+\mathcal{H}(\mathcal{P},\delta)\right\}$, $\alpha_{t}\leq\tau\leq t-1$. Due to this, we have $\sum_{\tau=\alpha_{t}}^{t-1}\Pr\{\mathcal{E}_{\delta,\tau}\}\preceq(t-\alpha_{t})\exp\left\{-2\zeta_{\delta}\left[\min_{\alpha_{t}\leq\tau\leq t}\mathcal{D}_{\tau}\right]^{2}N_{[\alpha_{t}:t]}+\mathcal{H}(\mathcal{P},\delta)\right\}$, where $N_{[\alpha_{t}:t]}\triangleq\min_{\alpha_{t}\leq\tau\leq t}w_{\tau}$. This completes the proof. $\blacksquare$

Appendix E Proof of Theorem 3

First, we upper bound $\mathbb{E}[\bar{p}_{k}(t)]$. As in [4], we consider the following instantaneous drift-plus-penalty expression denoted by $\mathcal{P}_{\tau,V}\triangleq\Delta(\tau+D)+Vp_{0}(\tau)$

[TABLE]

where $\mathcal{E}_{\delta,\tau}$ is the error in slot $\tau\in\mathbb{N}$ of step $1$ of the ADPP Algorithm due to incorrectly detecting the “right distribution,” $\mathcal{P}_{i^{*}}\in\mathcal{P}_{c}$ (see (13)). Next, we will compute an upper bound on the second term in (62), i.e., $\mathbb{E}\left[\mathcal{P}_{\tau,V}\left|\right.\mathcal{E}_{\delta,\tau}\right]$. Assume that the output of the Algorithm at time $\tau$ is $m_{\tau}^{*}$, and the corresponding induced probability be $\theta^{*}_{m}(\tau)=1$ if $m=m_{\tau}^{*}$, zero otherwise. Now, we consider the following drift-plus-penalty bound on the second term in (62) conditioned on $\mathbf{Q_{\tau}}$

[TABLE]

where $\rho\triangleq\sum_{k=1}^{K}(p_{\text{max},k}-c_{k})^{2}$, $\mathcal{C}_{k,\tau}\triangleq\left[\sum_{m=1}^{F}\theta^{*}_{m}(\tau)r_{k,\pi_{\tau}}^{(m)}-c_{k}\right]$, $H_{\tau}\triangleq(1+2D)B_{\tau}$, $B_{\tau}$ is as defined in (12). In the above, $(a)$ follows from Lemma $6$ of [4], and $(b)$ follows from the fact that $Q_{k}(\tau)\leq\tau(p_{\text{max},k}-c_{k})$, $\mathcal{C}_{k,\tau}\leq(p_{\text{max},k}-c_{k})$, and $p_{\text{max},k}$, $k=1,2,\ldots,K$ is the maximum cost/penalties. Recall that $\mathbf{Q_{\tau}}=\{Q_{1}(\tau),Q_{2}(\tau),\ldots,Q_{K}(\tau)\}$ and $Q_{k}(\tau+1)=\max\{Q_{k}(\tau)+p_{k}(\tau-D)-c_{k},0\},k=1,2,\ldots,K$. Since, $Q_{k}(0)=0$, $Q_{k}(1)=\max\{0+p_{k}(1-D)-c_{k},0\}\leq(p_{\text{max},k}-c_{k})$, and $Q_{k}(2)=\max\{Q_{k}(1)+p_{k}(2-D)-c_{k},0\}\leq\max\{(p_{\text{max},k}-c_{k})+p_{k}(2-D)-c_{k},0\}\leq 2(p_{\text{max},k}-c_{k})$. From the induction argument, we have $Q_{k}(\tau)\leq\tau(p_{\text{max},k}-c_{k})$, $k=1,2,\ldots,K$.. Taking the expectation of the above with respect to $\mathbf{Q_{\tau}}$ conditioned on $\mathcal{E}_{\delta,\tau}$ leads to the following result

[TABLE]

Applying Lemma $5$ of [4] to the first term in (62) conditioned on $\mathbf{Q_{\tau}}$, we get

[TABLE]

where $\mathcal{C}_{k,\tau}=\left[\sum_{m=1}^{F}\theta^{*}_{m}(\tau)r_{k,\pi_{\tau}}^{(m)}-c_{k}\right]$ is as before, $R_{0,\mathcal{P}_{i^{*}}}(\tau)\triangleq\sum_{m=1}^{F}\theta_{m}^{*}r_{0,\mathcal{P}_{i^{*}}}^{(m)}$, $R_{k}^{*}(\tau)\triangleq\sum_{m=1}^{F}\theta_{m}^{*}r_{k,\pi_{\tau}}^{(m)}$, $\Delta^{(\omega)}_{\pi_{\tau},\pi}\triangleq\left|\pi_{\tau}(\omega)-\pi(\omega)\right|$, $H_{\tau}\triangleq B_{\tau}(1+2D)$, and ${J}^{\pi}_{\pi_{\tau}}\triangleq\max_{0\leq k\leq K}p_{\text{max},k}\left(\|\pi_{\tau}-\pi\|_{1}+\|\mathcal{P}_{i^{*}}-\pi\|_{1}\right)$. Consider the following

[TABLE]

where $c_{k}^{{}^{\prime}}\triangleq c_{k}-{J}^{\pi}_{\pi_{\tau}}$. We need $c_{k}>{J}^{\pi}_{\pi_{\tau}}$. The above inequality is obtained by (a) adding and subtracting $\mathcal{P}_{i^{*}}$, (b) using triangle inequality, and (c) following the steps that lead to the first three terms in (65), and using the fact that $\|\mathcal{P}_{i^{*}}-\pi\|_{1}<\delta$. Substituting (66) in (65), we get

[TABLE]

Note that at each time slot $\tau$, the Algorithm chooses to minimize the right hand side of the above term when there is no error. Thus, choosing an alternative algorithm say $\theta_{m}$ will maximize the right hand side of (67). Towards bounding the above further, let us choose a $\theta_{m}$ denoted $\theta_{m,\texttt{opt}}^{{}^{\prime}}$ that optimally solves the problem $\mathbf{LP}_{\mathcal{P}_{i^{*}}}$ but with $c_{k}$ replaced by $c_{k}^{{}^{\prime}}$. Further, let the corresponding optimal cost be $p^{{}^{\prime}}_{\texttt{opt}}$. From Assumption 2, it follows that $p^{{}^{\prime}}_{\texttt{opt}}<p^{(\text{opt})}_{\mathcal{P}_{i^{*}}}+c{J}^{\pi}_{\pi_{\tau}}$. Using the optimal $\theta_{m,\texttt{opt}}^{{}^{\prime}}$ in (67), we get

[TABLE]

where the inequality $(a)$ is obtained by noting that for $\theta_{m}=\theta_{m,\texttt{opt}}^{{}^{\prime}}$, $\left[\sum_{m=1}^{F}\theta_{m,\texttt{opt}}^{{}^{\prime}}r_{k,\mathcal{P}_{i^{*}}}^{(m)}-c_{k}^{{}^{\prime}}\right]<0$, and $p^{{}^{\prime}}_{\texttt{opt}}=\sum_{m=1}^{F}\theta_{m,\texttt{opt}}^{{}^{\prime}}r_{0,\mathcal{P}_{i^{*}}}^{(m)}$. Using $p^{(\text{opt})}_{\mathcal{P}_{i^{*}}}<p^{\text{(opt)}}+(c+1)\Delta_{\pi,\mathcal{P}_{i^{*}}}$ from Theorem 1, we get

[TABLE]

where $\mathcal{\psi_{\texttt{const}}}\triangleq p^{\text{(opt)}}+(c+1)\Delta_{\pi,\mathcal{P}_{i^{*}}}$, and $\Delta_{\pi,\mathcal{P}_{i^{*}}}$ is as defined in Theorem 1. Now, taking the expectation with respect to $\mathbf{Q_{t}}$ conditioned on $\mathcal{E}^{c}_{\delta,\tau}$, we get

[TABLE]

Next, we borrow the results from Lemma 2 to obtain an upper bound (also, see (16)) on the probability of error in (62), i.e.,

**Result III: **

[TABLE]

where $q^{(\tau)}_{e,\texttt{up}}$ is as defined in Lemma 2. Using Result I, Result II and Result III in (62), we get

[TABLE]

where $\mathcal{K}\triangleq\left(H_{\tau}+Vp_{\text{max},0}\right)P^{(\tau)}_{e,\texttt{up}}$. Summing the above over all slots $\tau=0,1,2,\ldots,t-1$, and dividing by $t$, we get

[TABLE]

Using the fact that $\mathcal{L}(t+D)\geq 0$, and $L(D)\leq C$ for some constant $C>0$, and after rearranging the terms, we get

[TABLE]

where $\psi_{t}(\delta)$ is as defined in the theorem, and $\mathbb{E}[\bar{p}_{0}(t)]\triangleq\frac{1}{t}\sum_{\tau=0}^{t-1}\mathbb{E}p_{0}(\tau)$. For any $\epsilon>0$, choosing $\epsilon_{0}=(c+1)\Delta_{\pi,\mathcal{P}_{i^{*}}}+\psi_{t}(\delta)+\frac{{\alpha_{t}}{(p_{\text{max},k}-p_{\text{min},k})}}{t-\alpha_{t}}+\epsilon$ satisfies the bound on $\epsilon_{0}$ in Theorem 2. Again from Theorem 2 and the bound in (73), we have $\epsilon_{t,0}\triangleq\epsilon_{0}+p^{(opt)}-\frac{1}{t}\sum_{\tau=0}^{t-1}\mathbb{E}p_{0}(\tau)\geq\epsilon$. Thus, using $\epsilon$ in place of $\epsilon_{t,0}$ in (15), $v_{t}=(t-\alpha_{t})/u_{t}$, and substituting $\sum_{\tau=\alpha_{t}}^{t}P^{(\tau)}_{e,\texttt{up}}\leq(t-\alpha_{t})S_{t,\delta}$ from Lemma 2, we get the following upper bound

[TABLE]

It is easy to verify that the above is less than or equal to

[TABLE]

provided $t\in\mathcal{T}_{t,0}$, where $\mathcal{T}_{t,0}$ is as defined in the theorem. This proves the first part of the Theorem.

Multiplying (72) by $t$, substituting for $\mathcal{L}(t+D)$, and using the fact that for all time slots $\tau$, there exists a constant $F$ such that $F\geq p^{\text{(opt)}}-\mathbb{E}\left[p_{0}(\tau)\right]$, we get (refer to the proof of Theorem $3$ of [4] for details)

[TABLE]

where $\Gamma_{t}$ is as defined in the theorem. Using Jensen’s inequality, it follows from the above bound that

[TABLE]

for all $k=1,2,\ldots,K$. From Lemma $4$ of [4], we have $\mathbb{E}\{\bar{p}_{k}(t)\}\leq c_{k}+Q_{\texttt{up}}(t)$. Now, the right hand side of (15) for $\epsilon_{t,k}=\epsilon$, $\epsilon_{k}=Q_{\texttt{up}}(t)+\epsilon$ is less than or equal to $\gamma_{1}$ provided $t\in\mathcal{T}_{t,1}$, where $\mathcal{T}_{t,1}$ is as defined in the theorem. $\blacksquare$

Appendix F Proof of Theorem 4

Note that conditioned on $\mathcal{E}_{[\alpha_{t}:t]}^{c}$, $\mathbf{X}_{t-s}\longrightarrow\mathbf{Q}_{t}\longrightarrow\mathbf{X}_{t}$ forms a Markov chain (see Fig. 2). Thus, from[25], we have

[TABLE]

where $\eta_{ch_{1}}\triangleq\sup_{q\neq q^{{}^{\prime}}}\|\Pr\{\mathbf{X}_{t}\left|\right.\mathbf{Q}_{t}=q,\mathcal{E}_{[\alpha_{t}:t]}^{c}\}-\Pr\{\mathbf{X}_{t}\left|\right.\mathbf{Q}_{t}=q^{{}^{\prime}},\mathcal{E}_{[\alpha_{t}:t]}^{c}\}\|_{\texttt{TV}}$ is the Dobrushin’s contraction coefficient for the channel $\mathbf{Q}_{t}$ to $\mathbf{X}_{t}$. By letting $\mathcal{Q}_{t}$ to denote the set of all possible vectors that $\mathbf{Q}_{t}$ can take, we have that $q,q^{{}^{\prime}}\in\mathcal{Q}_{t}$. Further, as $t$ increases, the cardinality of $\mathcal{Q}_{t}$ grows. However, a “small” change in the queue will not effect the strategy used, and hence $\eta_{ch_{1}}=0$. We now make this observation more precise. As before, let $\mathcal{M}_{t}:\mathcal{Q}_{t}\rightarrow\{1,2,\ldots,F\}$ be the rule induced by the ADPP algorithm that determines the strategy given the queue at time $t$. Define $\mathbb{B}_{t}(m^{*})\triangleq\{\mathbf{Q}_{t}\in\mathcal{Q}_{t}:\mathcal{M}_{t}(\mathbf{Q}_{t})=m^{*}\}$. Note that $\mathbb{B}_{t}(m^{*})\bigcap\mathbb{B}_{t}(m)=\phi$, $m\neq m^{*}$. Using the above set, we have the following equivalence relation. We say that $q\sim q^{{}^{\prime}}$ if and only if $q,q^{{}^{\prime}}\in\mathbb{B}_{t}(m^{*})$ for some $m^{*}\in\{1,2,\ldots,F\}$, and $q\nsim q^{{}^{\prime}}$ otherwise. It is easy to see that if $q\sim q^{{}^{\prime}}$, then $\|\Pr\{\mathbf{X}_{t}\left|\right.\mathbf{Q}_{t}=q,\mathcal{E}_{[\alpha_{t}:t]}^{c}\}-\Pr\{\mathbf{X}_{t}\left|\right.\mathbf{Q}_{t}=q^{{}^{\prime}},\mathcal{E}_{[\alpha_{t}:t]}^{c}\}\|_{\texttt{TV}}=0$. Using this fact, we can equivalently write $\eta_{ch_{1}}$ as $\eta_{ch_{1}}=\sup_{q\nsim q^{{}^{\prime}}}\|\Pr\{\mathbf{X}_{t}\left|\right.\mathbf{Q}_{t}=q,\mathcal{E}_{[\alpha_{t}:t]}^{c}\}-\Pr\{\mathbf{X}_{t}\left|\right.\mathbf{Q}_{t}=q^{{}^{\prime}},\mathcal{E}_{[\alpha_{t}:t]}^{c}\}\|_{\texttt{TV}}$. When $q\nsim q^{{}^{\prime}}$, we have $m\triangleq\mathcal{M}_{t}(q)\neq\mathcal{M}_{t}(q^{{}^{\prime}})\triangleq m^{{}^{\prime}}$; this leads to the following

[TABLE]

where the last inequality above follows from Assumption 4. Further, since $\kappa<\log 3$, we have $\frac{(e^{\kappa}-1)}{2}<1$. Using the fact that $\mathbf{X}_{t-s}\longrightarrow(\mathbf{Q}_{t-1},\mathbf{X}_{t-1})\longrightarrow\mathbf{Q}_{t}$ forms a Markov chain (see Fig. 2), we can further bound the right hand side of (77) as follows

[TABLE]

where the Dobrushin’s contraction coefficient for the channel $(\mathbf{Q}_{t-1},\mathbf{X}_{t-1})$ to $(\mathbf{Q}_{t-2},\mathbf{X}_{t-2})$ conditioned on $\mathcal{E}_{[\alpha_{t}:t]}^{c}$ is given by

[TABLE]

In the above, $\mathcal{V}_{a,b,t}\triangleq\{\mathbf{X}_{t-2}=a,\mathbf{Q}_{t-2}=b,\mathcal{E}_{[\alpha_{t}:t]}^{c}\}$, $a\in\{p,p^{{}^{\prime}}\}$ and $b\in\{q,q^{{}^{\prime}}\}$. Further, $(a)$ and $(b)$ follow from the data processing inequality and SDPI for the channel $(\mathbf{Q}_{t-1},\mathbf{X}_{t-1})$ to $(\mathbf{Q}_{t-2},\mathbf{X}_{t-2})$, respectively [25]. Note that conditioned on $\mathcal{E}_{[\alpha_{t}:t]}^{c}$, the Dobrushin’s contraction coefficients remain the same for all $(\mathbf{Q}_{t-i},\mathbf{X}_{t-i})$ to $(\mathbf{Q}_{t-i-1},\mathbf{X}_{t-i-1})$, $i=1,2,\ldots,s$. Using this argument and applying the SDPI repeatedly for (79), we get

[TABLE]

where $\mathcal{Z}_{s,t}\triangleq(\mathbf{Q}_{t-s+1},\mathbf{X}_{t-s+1})$, and the last inequality follows from the fact that $I(\mathbf{Q}_{t-s+1},\mathbf{X}_{t-s+1};\mathbf{X}_{t-s}\left|\right.\mathcal{E}_{[\alpha_{t}:t]}^{c})\leq H(\mathbf{X}_{t-s}\left|\right.\mathcal{E}_{[\alpha_{t}:t]}^{c})\leq H(\mathbf{X}_{t-s})\leq\log(N_{s})$, where $N_{s}=F\left|\Omega\right|(K+1)$ is the maximum number of possible values that $\mathbf{X}_{\tau}$ can take for all $\tau\in\mathbb{N}$. Using the bound in (81), and (78) in (77), we get

[TABLE]

Now, it remains to bound $\eta_{{ch}_{2}}$. Towards this, consider

[TABLE]

where $\mathcal{V}_{p,q,t}$ is as defined earlier, $\sigma_{x,y}\triangleq\max\{x+y-\mathbf{C},0\}$, $x\in\{p,p^{{}^{\prime}}\}$, $y\in\{q,q^{{}^{\prime}}\}$, and $\delta(.,.)$ is the Kroneckar delta function. In the above, $(a)$ is obtained by using the Bayes rule followed by the fact that $\mathbf{X}_{t-1}$ is independent of $\mathcal{V}_{p,q,t}$ and $\mathcal{V}_{p^{{}^{\prime}},q^{{}^{\prime}},t}$ conditioned on $\mathbf{Q}_{t-1}$ and $\mathcal{E}_{[\alpha_{t}:t]}^{c}$, and $(b)$ follows because $\mathbf{Q}_{t-1}$ is a deterministic function of $\mathbf{Q}_{t-2}$ and $\mathbf{X}_{\neq 0,t-2}$, i.e., $\mathbf{Q}_{t-1}=\max\{\mathbf{Q}_{t-2}+\mathbf{X}_{t-2}-\mathbf{C},0\}$. Now, we have

[TABLE]

where the last inequality above follows from the same argument that was used to obtain the bound on $\eta_{{ch_{1}}}$ in (78). Using (84) in (82), we get

[TABLE]

where $\theta\triangleq\max\left\{\frac{(e^{\kappa}-1)}{2},\frac{1}{2}\right\}$. Substituting the above in (22), we get the desired result in the Theorem. This completes the proof. $\blacksquare$

Appendix G Proof of Lemma 3

Since $u_{t}v_{t}=t-\alpha_{t}$, it is possible to choose $\alpha_{t}=\mathcal{O}(\sqrt{t})$, ${u_{t}}=\mathcal{O}(\sqrt{t})$, and $v_{t}=\mathcal{O}(\sqrt{t})$. Also, let $w_{t}=\mathcal{O}(\sqrt{t})$, and thus, we have

[TABLE]

as $t\rightarrow\infty$. Further,

[TABLE]

$k=0,1,2,\ldots,K,$ as $t\rightarrow\infty$. Using $w_{t}=\mathcal{O}(\sqrt{t})$, and $\mathcal{H}(\mathcal{P},\delta)<\infty$ in the expression for $P_{e,\texttt{up}}^{(\tau)}$, we have

[TABLE]

By letting $V=\mathcal{O}(\sqrt{t})$ and the fact that $\sum_{\tau=0}^{\infty}P_{e,\texttt{up}}^{(\tau)}<\infty$, we have

[TABLE]

Since $F<\infty$, and $V=\mathcal{O}(\sqrt{t})$, it follows that $\lim_{t\rightarrow\infty}Q_{\texttt{up}}(t)=\lim_{t\rightarrow\infty}\left[\sqrt{\frac{F}{\sqrt{t}}+\frac{\Gamma_{t}}{t^{2}}}\right]=0$. Using this along with (88), (86) and (87) in Theorem 3, we get the desired result. $\blacksquare$

Appendix H Proof of Theorem 5

From the Borel-Cantelli Lemma, it follows that if

[TABLE]

then $\lim_{t\rightarrow\infty}\frac{1}{t}\sum_{\tau=0}^{t-1}p_{k}(\tau)-c_{k}\leq\epsilon_{k}$ almost surely [24]. Using $u_{t}=\mathcal{O}(\sqrt{t})$, $v_{t}=\mathcal{O}(\sqrt{t})$, $\alpha_{t}=\mathcal{O}(\sqrt{t})$, and an upper bound on the above from Theorem 2, it suffices to show that

[TABLE]

where $\bar{\epsilon}_{t,k}=\frac{t\epsilon_{t,k}-{\alpha_{t}}(p_{\text{max},k}-p_{\text{min},k})}{t-\alpha_{t}}$. Note that as $t\rightarrow\infty$, $\bar{\epsilon}_{t,k}\rightarrow\epsilon_{t,k}$. By choosing $u_{t}=\mathcal{O}(\sqrt{t})$ and $v_{t}=\mathcal{O}(\sqrt{t})$ as in the Theorem, it is easy to see that the first term above is finite since the summand is a product of a $\mathcal{O}({\sqrt{t}})$ term and an exponentially decreasing function of $t$. Using the result of Lemma 2 with $w_{t}=\mathcal{O}(\sqrt{t})$, it is easy to see that there exists a $t^{*}<\infty$ such that $t^{*}>D+w_{t^{*}}-1$. Thus, for all $\tau<t^{*}$, $\Pr\{\mathcal{E}_{\delta,\tau}\}=1/M_{\delta}$. Using this, the second term above can be written as

[TABLE]

where $\phi_{\tau,t,\delta}\triangleq 2\zeta_{\delta}\left[\min_{\alpha_{t}\leq\tau\leq t}\mathcal{D}_{\tau}\right]^{2}N_{[\alpha_{t}:t]}$, $N_{[\alpha_{t}:t]}\triangleq\min_{\alpha_{t}\leq\tau\leq t}w_{\tau}$ as in Lemma 2. Since $t^{*}$ is finite, we can say that the first term in the second inequality is bounded. Since $\alpha_{t}=\mathcal{O}(\sqrt{t})$, $(t-\alpha_{t})\exp\left\{-{\phi_{\tau,t,\delta}}+\mathcal{H}(\mathcal{P},\delta)\right\}\rightarrow 0$ exponentially fast as $t\rightarrow\infty$. Hence, the second inequality is also bounded. Now, using the result from (26) of Corollary 1, the third term in (90) becomes

[TABLE]

which is finite since $\theta<1$ (see Corollary 1). Thus, we have $p_{k}(\tau)-c_{k}\leq\epsilon_{k}$ almost surely for all $k$. Next, we need to show that the first term in (H) is finite. Towards this it suffices to show that $\bar{\epsilon}_{t,k}<\infty$ since $u_{t}=\mathcal{O}(\sqrt{t})$ and $v_{t}=\mathcal{O}(\sqrt{t})$. Equivalently, from the definition of $\bar{\epsilon}_{t,k}$ in Theorem 2, we need to show that $\lim_{t\rightarrow\infty}{\epsilon}_{t,k}<\infty$. From the proof of Theorem 3, using $\infty>\epsilon_{0}>\frac{1}{t}\sum_{\tau=0}^{t-1}\mathbb{E}p_{0}(\tau)-p^{\text{(opt)}}+\frac{{\alpha_{t}}{(p_{0,\texttt{max}}-p_{0,\texttt{min}})}}{t-\alpha_{t}}>(c+1)\Delta_{\pi,\mathcal{P}_{i^{*}}}+\psi_{t}(\delta)+\frac{{\alpha_{t}}{(p_{0,\texttt{max}}-p_{0,\texttt{min}})}}{t-\alpha_{t}}+\epsilon$ and $\epsilon_{k}=Q_{\texttt{up}}(t)+\epsilon$, we get the desired result. $\blacksquare$

Appendix I Proof of Lemma 4

Consider the following expression from the definition of $\eta_{\texttt{ch}_{1}}$ in (IV-B) with $\gamma\triangleq\{\gamma_{D},\gamma_{D-2},\ldots,\gamma_{1}\}$.

[TABLE]

where $\mathcal{B}_{1}\triangleq\{\mathbf{Q}_{t}=\gamma_{D},\mathbf{Q}_{t-1}=\gamma_{D-1}\}$, $\mathcal{B}_{2}\triangleq\{\mathbf{Q}_{t}=\gamma_{D},\mathbf{Q}_{t-1}=\gamma_{D-1},\ldots,\mathbf{Q}_{t-D+1}=\gamma_{1}\}$, and the last step follows from the Markov chain property, i.e., conditioned on $\mathbf{Q}_{t-i}$, $\mathbf{X}_{t-i}$ is independent of $\mathbf{Q}_{t-i+j}$, $j=0,1,\ldots,i$. As in the proof of Theorem 4, we let $\mathcal{Q}_{t}$ to denote the set of all possible vectors that $\mathbf{Q}_{t}$ can take. Further, as $t$ increases, the cardinality of $\mathbf{Q}_{t}$ grows. However, a “small” change in the queue will not effect the strategy used, and hence $\eta_{\texttt{ch}_{1}}=0$. We now make this observation more precise. Let $\mathcal{M}_{t}:\mathcal{Q}_{t}\rightarrow\{1,2,\ldots,F\}$ be the rule induced by the ADPP algorithm that determines the strategy given the queue at time $t$. For any strategy $m^{*}$, define $\mathbb{B}_{t}(m^{*})\triangleq\{\mathbf{Q}_{t}\in\mathcal{Q}_{t}:\mathcal{M}_{t}(\mathbf{Q}_{t})=m^{*}\}$. Using the above set, we have the following equivalence relation. We say that $q\sim q^{{}^{\prime}}$ if and only if $q,q^{{}^{\prime}}\in\mathbb{B}_{t}(m^{*})$ for some $m^{*}\in\{1,2,\ldots,F\}$, and $q\nsim q^{{}^{\prime}}$ otherwise. It is easy to see that if $q\sim q^{{}^{\prime}}$, then $\|\Pr\{\mathbf{X}_{t-i}\left|\right.\mathbf{Q}_{t-i}=q,\mathcal{E}_{[\alpha_{t}:t]}^{c}\}-\Pr\{\mathbf{X}_{t-i}\left|\right.\mathbf{Q}_{t-i}=q^{{}^{\prime}},\mathcal{E}_{[\alpha_{t}:t]}^{c}\}\|_{\texttt{TV}}=0$, $i=0,1,\ldots,D-1$. Using this fact, we can equivalently write $\eta_{\texttt{ch}_{1}}$ as $\eta_{\texttt{ch}_{1}}=\sup_{q\nsim q^{{}^{\prime}}}\|\Pr\{\mathbf{X}_{t}\left|\right.\mathbf{Q}_{t}=q,\mathcal{E}_{[\alpha_{t}:t]}^{c}\}-\Pr\{\mathbf{X}_{t}\left|\right.\mathbf{Q}_{t}=q^{{}^{\prime}},\mathcal{E}_{[\alpha_{t}:t]}^{c}\}\|_{\texttt{TV}}$. However, when $q\nsim q^{{}^{\prime}}$, we have $\mathcal{M}_{t}(q)\neq\mathcal{M}_{t}(q^{{}^{\prime}})$; using this and (93) in the definition of $\eta_{\texttt{ch}_{1}}$ leads to the following

[TABLE]

where $(a)$ follows by substituting (93) in the definition of $\eta_{\texttt{ch}_{1}}$, and $(b)$ follows from the Assumption $4$ and the definition of the total variational norm. This completes the proof. $\blacksquare$

Appendix J Proof of Lemma 5

Note that $\mathbf{Q}_{j-1,t}$ is a deterministic function of $\mathbf{X}_{j,t}$ and $\mathbf{Q}_{j,t}$, i.e., $\mathbf{Q}_{j-1,t}=\max\left\{\mathbf{Q}_{j,t}+\mathbf{X}_{j,t}-\mathbf{C},0\right\}$, where $\mathbf{C}\triangleq\{\underbrace{c_{1},\ldots,c_{1}}_{{D}\text{ times}},\underbrace{c_{2},\ldots,c_{2}}_{{D}\text{ times}},\ldots,\underbrace{c_{K},\ldots,c_{K}}_{{D}\text{ times}}\}$. Using (38), $\eta_{ch_{j}}$ can be written as

[TABLE]

where $\sigma_{x,y}\triangleq\max\{x+y-\mathbf{C},0\}$, $x\in\{p,p^{{}^{\prime}}\}$, $y\in\{q,q^{{}^{\prime}}\}$, and $\delta(.,.)$ is the Kroneckar delta function as in Theorem 4 and $(a)$ follows from the fact that $\mathbf{Q}_{j-1,t}$ is a deterministic function of $\mathbf{X}_{j,t}$ and $\mathbf{Q}_{j,t}$, i.e., $\mathbf{Q}_{j-1,t}=\max\left\{\mathbf{Q}_{j,t}+\mathbf{X}_{j,t}-\mathbf{C},0\right\}$. The last inequality follows from the argument used in the proof of Lemma 4. $\blacksquare$