Finite-Time Performance of Distributed Temporal Difference Learning with Linear Function Approximation

TL;DR

This paper analyzes the finite-time convergence of a distributed temporal difference learning algorithm with linear function approximation in multi-agent reinforcement learning, considering the effects of network topology, discount factor, and Markov chain mixing time.

Contribution

It provides the first finite-sample analysis of distributed TD(λ) with explicit bounds depending on key parameters, including network structure and Markov chain properties.

Findings

Explicit convergence rate bounds as a function of network topology and mixing time.

Mathematical explanation for the impact of λ on approximation accuracy and convergence speed.

Trade-off analysis between accuracy and speed based on λ choice.

Abstract

We study the policy evaluation problem in multi-agent reinforcement learning, modeled by a Markov decision process. In this problem, the agents operate in a common environment under a fixed control policy, working together to discover the value (global discounted accumulative reward) associated with each environmental state. Over a series of time steps, the agents act, get rewarded, update their local estimate of the value function, then communicate with their neighbors. The local update at each agent can be interpreted as a distributed variant of the popular temporal difference learning methods {\sf TD}$ (\lambda)$. Our main contribution is to provide a finite-analysis on the performance of this distributed {\sf TD}$(\lambda)$ algorithm for both constant and time-varying step sizes. The key idea in our analysis is to use the geometric mixing time $\tau$ of the underlying Markov chain, that is, although the "noise" in our algorithm is Markovian, its dependence is very weak at samples spaced out at every $\tau$. We provide an explicit upper bound on the convergence rate of the proposed method as a function of the network topology, the discount factor, the constant $\lambda$, and the mixing time $\tau$. Our results also provide a mathematical explanation for observations that have appeared previously in the literature about the choice of $\lambda$. Our upper bound illustrates the trade-off between approximation accuracy and convergence speed implicit in the choice of $\lambda$. When $\lambda=1$, the solution will correspond to the best possible approximation of the value function, while choosing $\lambda = 0$ leads to faster convergence when the noise in the algorithm has large variance.

Equations

$$\displaystyle J(i)\triangleq\mathbb{E}\left[\sum_{k=0}^{\infty}\frac{\gamma^{k}}{N}\sum_{v\in{\cal V}}{\cal R}^{v}(s_{k},u_{k},s_{k+1})\Bigm{|}s_{0}=i\right],$$

$$J(i)=\sum _{j=1}^n{p}_{ij}\left\{\frac{1}{N}\sum _{v\in \mathcal{V}}{\mathcal{R}}^v(i,j)+\gamma J(j)\right\},i\in \mathcal{S}.$$

$$\tilde{J}(i,\theta )=\sum _{\ell =1}^L{\theta }_{\ell }{\varphi }_{\ell }(i),$$

$$\varphi (i)=({\varphi }_1(i),\dots ,{\varphi }_L(i){)}^T\in {\mathbb{R}}^L.$$

$$\displaystyle\Phi=\left[\begin{array}[]{ccc}|&&|\\ \phi_{1}&\cdots&\phi_{L}\\ |&&|\end{array}\right]=\left[\begin{array}[]{ccc}\mbox{---}&\phi(1)^{T}&\mbox{---}\\ \cdots&\cdots&\cdots\\ \mbox{---}&\phi(S)^{T}&\mbox{---}\end{array}\right]\in\mathbb{R}^{S\times L}.$$

$$\nabla \tilde{J}(i,\theta )=\varphi (i),\text{and}\nabla \tilde{J}={\Phi }^T.$$

$$y_k^v$$

$$d_k^v$$

$${\theta }_{k+1}^v$$

$$z_k^v=\sum _{u=0}^k(\gamma \lambda {)}^{k-u}\varphi (s_u).$$

$${\hat{\theta }}_k^v=\frac{{\sum }_{t=0}^k{\alpha }_t{\theta }_k^v}{{\sum }_{t=0}^k{\alpha }_t}\cdot$$

$$\begin{array}{rl}{y}_k^v& =\sum _{u\in {\mathcal{N}}_v}{W}_{vu}{\theta }_k^u\\ d_k^v& =r_k^v+{\left(\gamma \varphi (s_{k+1})-\varphi (s_k)\right)}^T{\theta }_k^v\\ {\theta }_{k+1}^v& =y_k^v+{\alpha }_k{d}_k^v{z}_k^v\\ z_{k+1}^v& =\gamma \lambda z_k^v+\varphi (s_{k+1})\end{array}$$

$$S_{k+1}^v$$

$${\hat{\theta }}_{k+1}^v$$

$$\Pi J=\underset{y\in \text{span}\{{\varphi }_{\ell }\}}{\mathrm{argmin}}\Vert y-J{\Vert }_{\mathbf{D}},$$

$$\mathbf{U}$$

$$b^v$$

$$\mathbf{A}{\theta }^{\ast }=b\triangleq \frac{1}{N}\sum _{v\in \mathcal{V}}{b}^v,$$

$$\mathbf{A}={\Phi }^T\mathbf{D}(\mathbf{U}-\mathbf{I})\Phi \in {\mathbb{R}}^{L\times L}.$$

$$\Vert \Pi J-J{\Vert }_D\le \Vert \Phi {\theta }^{\ast }-J{\Vert }_D\le \frac{1-\gamma \lambda }{1-\gamma }\Vert \Pi J-J{\Vert }_D.$$

$$\begin{array}{rl}\mathbf{A}(X_k)& =z_k(\gamma \varphi (s_{k+1})-\varphi (s_k){)}^T,\\ b^v(X_k)& =r_k^v{z}_k.\end{array}$$

$${\theta }_{k+1}^v=\sum _{u\in {\mathcal{N}}_v}{W}_{vu}{\theta }_k^u+{\alpha }_k(\mathbf{A}(X_k){\theta }_k^v+b^v(X_k))$$

$$\displaystyle{\bar{\theta}}_{k+1}={\bar{\theta}}_{k}+\alpha_{k}\big{(}{\bf A}(X_{k}){\bar{\theta}}_{k}+{\bar{b}}(X_{k})\big{)},$$

$$\bar{b}(X_k)=\frac{1}{N}\sum _{v\in \mathcal{V}}{b}^v(X_k)=\frac{1}{N}\sum _{v\in \mathcal{V}}{r}_k^v{z}_k\in {\mathbb{R}}^L.$$

$$\displaystyle\Theta\triangleq\left[\begin{array}[]{ccc}\mbox{---}&\left[\theta^{1}\right]^{T}&\mbox{---}\\ \cdots&\cdots&\cdots\\ \mbox{---}&\left[\theta^{N}\right]^{T}&\mbox{---}\end{array}\right]\in\mathbb{R}^{N\times L},\quad{\bf B}(X_{k})\triangleq\left[\begin{array}[]{ccc}\mbox{---}&\left[b^{1}(X_{k})\right]^{T}&\mbox{---}\\ \cdots&\cdots&\cdots\\ \mbox{---}&\left[b^{N}(X_{k})\right]^{T}&\mbox{---}\end{array}\right]\in\mathbb{R}^{N\times L},$$

$${\Theta }_{k+1}=\mathbf{W}{\Theta }_k+{\alpha }_k{\Theta }_k\mathbf{A}(X_k{)}^T+{\alpha }_k\mathbf{B}(X_k).$$

$$\underset{k\to \infty }{\lim }\mathbb{E}[\mathbf{A}(X_k)]=\mathbf{A}\text{ and }\underset{k\to \infty }{\lim }\mathbb{E}[\bar{b}(X_k)]=b.$$

$$\begin{array}{rl}& \Vert \mathbb{E}[\mathbf{A}(X_k)-\mathbf{A}|X_0=X]\Vert \le \alpha ,\forall X,\forall k\ge \tau (\alpha )\\ & \Vert \mathbb{E}[\bar{b}(X_k)-b|X_0=X]\Vert \le \alpha ,\forall X,\forall k\ge \tau (\alpha ),\end{array}$$

$$\tau (\alpha )=C\log \left(\frac{1}{\alpha }\right).$$

$$\begin{array}{rl}\Vert \mathbf{A}(X_k)\Vert & \le \Vert z_k(\gamma \varphi (s_{k+1})-\varphi (s_k){)}^T\Vert \le (1+\gamma )\sum _{u=0}^t(\gamma \lambda {)}^{t-u}\le \frac{1+\gamma }{1-\gamma \lambda }\\ \Vert b^v(X_k)\Vert & \le \Vert r_k^v{z}_k\Vert \le \frac{R}{1-\gamma \lambda }\cdot \end{array}$$

$$\Vert \mathbf{A}\Vert \le \frac{1+\gamma }{1-\gamma \lambda }\text{and}\Vert b\Vert \le \frac{R}{1-\gamma \lambda },$$

Full text

Over a series of time steps, the agents act, get rewarded, update their local estimate of the value function, then communicate with their neighbors. The local update at each agent can be interpreted as a distributed variant of the popular temporal difference learning methods TD$(\lambda)$.

Our main contribution is to provide a finite-analysis on the performance of this distributed TD$(\lambda)$ algorithm for both constant and time-varying step sizes. The key idea in our analysis is to use the geometric mixing time $\tau$ of the underlying Markov chain, that is, although the “noise” in our algorithm is Markovian, its dependence is very weak at samples spaced out at every $\tau$. We provide an explicit upper bound on the convergence rate of the proposed method as a function of the network topology, the discount factor, the constant $\lambda$, and the mixing time $\tau$.

Our results also provide a mathematical explanation for observations that have appeared previously in the literature about the choice of $\lambda$. Our upper bound illustrates the trade-off between approximation accuracy and convergence speed implicit in the choice of $\lambda$. When $\lambda=1$, the solution will correspond to the best possible approximation of the value function, while choosing $\lambda=0$ leads to faster convergence when the noise in the algorithm has large variance.

1 Introduction

Reinforcement learning (RL) is a general paradigm for learning optimal policies in stochastic control problems based on simulation [1, 2, 3]. Relatively simple RL algorithms have been remarkably effective in solving challenging practical problems; notable examples include autonomous driving [4], robotics [5], helicopter flight [6], board games [7], and power networks [8]. In these applications, an agent tries to find an optimal policy (a mapping from state to action) through interactions with the environment, modeled as a Markov Decision Process (MDP), with the goal of optimizing its long-term future reward (or cost). One of the central problems in RL, and the one considered in this paper, is the policy evaluation problem, where the expected long-term reward for a given stationary policy is estimated. While interesting in its own right, policy evaluation also arises as a subproblem in RL policy search methods, including policy iteration and actor-critic methods [1, 2]. Temporal-difference learning (TD$(\lambda)$), originally proposed by Sutton [9], is one of the most efficient and practical methods for policy evaluation. The iterations for TD$(\lambda)$, discussed in detail in Section 2.1 below, are relatively simple, and can be implemented in an online fashion. The long-term future cost is estimated as a function of the current state, with the scalar parameter $\lambda\in[0,1]$ controlling the trade-off between the accuracy of the approximation and the susceptibility to simulation noise. The convergence of TD$(\lambda)$ is a classical result, and has been analyzed under various sets of assumptions in [10, 11, 12, 13, 14]. In practice, the domain of the policy value function (the state space) can be very large. To avoid (or at least mitigate) the “curse of dimensionality”, the value function is chosen from a parametric set [15, 16, 7]. The particular case we analyze here is linear function approximation, where the value function at each state is a linear combination of (fixed) feature vectors, which allows us to explicitly characterize its rate of convergence.

In this paper, we study the policy evaluation problem where multiple agents explore a common environment. We are motivated by broad applications of the multi-agent paradigm within engineering, for example, mobile sensor networks [17, 18], cell networks [19], and power networks [8]. In this multi-agent reinforcement learning (MARL) setting, each agent takes its own action based on the current (global) state, and the agents receive different local rewards that are a function of their current state, their new state, and their action. We assume that each agent observes only its own reward. Their goal is to cooperatively evaluate the global accumulative reward while only communicating with a small subset of the other agents. We introduce a distributed variant of TD$(\lambda)$ algorithm with linear function approximation, and consider data samples generated by the Markov process representing the environment. Thus, the noise in our algorithm is dependent since it is Markovian. Our results provide a finite-time analysis of distributed MARL TD$(\lambda)$, giving an upper bound on the distance of the current estimate from the eventual point of convergence as a function of number of iterations (and other problem parameters). The results in this paper generalize our preceding work [20], where we studied distributed TD$(0)$ under i.i.d sampling assumption (i.e, the noise in the algorithm is i.i.d).

1.1 Existing literature

In general, TD$(\lambda)$ with linear function approximation can be viewed as a variant of the celebrated stochastic approximation (SA) method, whose asymptotic convergence is typically analyzed using the so-called ordinary differential equation (ODE) method [21]. The ODE method shows that under the right conditions, the effects of noise eventually average out and the SA iterates asymptotically follow the trajectory of a stable ODE. In particular, Tsitsiklis and Van Roy considered a policy evaluation problem on a discounted MDP for both finite and infinite state spaces with linear function approximation [13]. By viewing TD as a stochastic approximation for solving a suitable Bellman equation, they characterized its almost-sure convergence using the ODE approach. Following this work, Borkar and Meyn provided a general and unified framework for the convergence of SA with broad applications in RL [22]. General results in this area can be found in the monograph by Borkar [21].

While the ODE method can be used to study the asymptotic convergence of TD$(\lambda)$, it is not obvious how to derive a rate of convergence using this approach. In general, TD$(\lambda)$ does not correspond to stochastic gradient descent (SGD) on any static optimization problem, making it challenging to characterize the consistency and quantify the progress of this method. Indeed, the convergence rates of TD$(\lambda)$ remained largely open until recently [23, 24, 25, 26, 27, 28, 29, 30]. In particular, a concentration bound was given in [23] for the SA algorithm under a strict stability assumption of the iterates. A finite-time analysis of the TD method with linear function approximation was simultaneously studied in [24, 25, 26, 27] for a single agent. These works carefully characterize the progress of the TD update and derive its convergence rate using standard techniques from the study of SGD and the results in [13]. Recently, the work in [28] provides a finite-time error bound of linear SA under Markovian noise and constant step sizes under very general conditions (e.g., without requiring an additional projection step to keep the iterates bounded). Motivated by this work, the authors in [29] study the finite-time performance of the linear SA using control theoretic approach, while the work in [30] provides the finite-time convergence of nonlinear SA.

Within the context of MARL, an asymptotic convergence of a distributed gossiping version of TD$(0)$ with linear function approximation was probably first studied in [31], where again convergence is established using the ODE approach. Similar results were also studied implicitly in [32], which analyzes the convergence of distributed actor-critic methods. While finite-time (convergence rate) analysis exists for single agent problems [24, 25, 26, 28, 29], a rate of convergence of distributed TD$(\lambda)$ does not exists in the current literature, and is the focus of this paper. While the convergence rates of distributed TD$(0)$ were studied in our earlier work [20], the results were derived under (perhaps unrealistically strong) assumptions on the noise being independent, and the algorithm included a projection step at each iterate that required a priori knowledge about the solution. Below, we study the rate of convergence of TD$(\lambda)$ under Markovian noise and without requiring any projection step. Our approach is motivated by the recent work in [28] about the rates of TD$(\lambda)$ for a single agent problem.

Finally, we mention some related RL methods for solving policy evaluation problems in both single agent RL and MARL, such as, the gradient temporal difference methods studied in [33, 34, 35, 36, 37, 38], least squares temporal difference (LSTD) [39, 40], and least squares policy evaluation (LSPE) [41, 42, 43]. Although they share some similarity with TD learning, these methods belong to a different class of algorithms whose update iterations are more complex. Our analysis focuses on a decentralized extension of TD online learning that is practical and simple to implement (although, as we will see below, not as straightforward to analyze).

1.2 Main contributions

In this paper, we study a distributed variant of the temporal difference learning method for solving the policy evaluation problem in multi-agent reinforcement learning. Our distributed algorithm is composed of a consensus step amongst locally communicating agents, followed by a local TD$(\lambda)$ update. Our main contribution is to provide a finite-time analysis on the performance of this distributed TD algorithm for both constant and time-varying step sizes. The key idea in our analysis is to properly utilize the geometric mixing time $\tau$ of the underlying Markov chain, which allows a systematic treatment of the Markovian “noise”, as its dependence becomes quantifiably weak for observations $\tau$ steps apart.

We provide an explicit upper bound on the rate of convergence of the proposed method as a function of the network topology, discount factor, the constant $\lambda$, and the mixing time $\tau$ of the underlying Markov chain. Our results theoretically address some numerical observations of TD$(\lambda)$ , that is, $\lambda=1$ gives the best approximation of the function values while $\lambda=0$ leads to better performance when there is large variance in the algorithm.

2 Multi-agent reinforcement learning

We consider a multi-agent reinforcement learning system of $N$ agents operating in a common environment, modeled by a Markov decision process. We assume that the agents can communicate with each other through a connected and undirected graph ${\cal G}=({\cal V},{\cal E})$, where ${\cal V}=\{1,\ldots,N\}$ and ${\cal E}={\cal V}\times{\cal V}$ are the vertex and edge sets, respectively. At each time step, the agents observe the state of the environment, take an action based on this observation, and receive a corresponding award. The goal of the agents is to cooperatively estimate the global discounted accumulative reward, which is composed of local rewards received by the agents. This is the policy evaluation problem for multi-agent systems, which can be mathematically characterized as follows.

Let ${\cal S}$ be the global finite state space of the environment, where $S$ denotes its cardinality. In addition, we denote by ${\cal U}^{v}$ the set of control actions at each agent $v$. At each time step $k\geq 0$, each agent $v\in{\cal V}$ observes the state $s_{k}\in{\cal S}$ of the environment and based on that observation take an action $u_{k}^{v}=\mu^{v}(s_{k})\in{\cal U}^{v}$. Here $\mu^{v}:{\cal S}\rightarrow{\cal U}^{v}$, the policy of agent $v$, is a function mapping the state of the environment to a control action in ${\cal U}^{v}$. We denote by ${\cal U}={\cal U}^{1}\times\ldots\times{\cal U}^{N}$ and let $u=u^{1}\times\ldots\times u^{N}\in{\cal U}$ be the joint action of the agents. As a consequence of their joint actions, the environment moves to a new state $s_{k+1}\in{\cal S}$. In addition, each agent $v$ receives an instantaneous local reward ${\cal R}^{v}_{k}={\cal R}^{v}(s_{k},u_{k},s_{k+1})$. The goal of the agents is to cooperatively find the average of total discounted accumulative reward $J$ over the network defined as

[TABLE]

where $\gamma$ is the discount factor.

The system evolves under a fixed, stationary policy at each agent, and so the dynamics of the environment can be modeled by a Markov chain. We use ${\bf P}$ for the transition probabilities for this underlying Markov chain: ${\bf P}(s_{k}=i,s_{k+1}=j)=p_{ij}$ for $i,j\in{\cal S}$ is the probability of the environment transitioning to state $j$ from state $i$. (These transition probabilities also of course depend on the policy, but since the policy remains fixed, we do not include it in the notation.) It is well-known that $J$ satisfies the Bellman equation [2, 1],

[TABLE]

We are interested in the case when the number of states is very large, and so computing $J$ exactly may be intractable. To mitigate this, we use low-dimensional approximation $\tilde{J}$ of $J$, restricting $\tilde{J}$ to be in a linear subspace. While more advanced nonlinear approximations using, for example, neural nets as in the recent works [16, 7] may lead to more powerful approximations, the simplicity of the linear model allows us to analyze it in detail. The linear function approximation $\tilde{J}$ is parameterized by a weight vector $\theta\in\mathbb{R}^{L}$, with

[TABLE]

for a given set of $L$ basis vectors $\phi_{\ell}:{\cal S}\rightarrow\mathbb{R}$, $\ell\in\{1,\ldots,L\}$. We are interested in the case $L\ll S$. Let $\phi(i)$ be the feature vector defined as111We refer to $\phi_{\ell}\in\mathbb{R}^{S}$ as the basis vectors and $\phi(i)\in\mathbb{R}^{L}$ as the feature vectors since it is defined on the environmental states.

[TABLE]

And let $\Phi\in\mathbb{R}^{S\times L}$ be a matrix, whose $i$-th row is the row vector $\phi(i)^{T}$ and whose $\ell$-th column is the vector $\phi_{\ell}=(\phi_{\ell}(1),\ldots,\phi_{\ell}(S))^{T}\in\mathbb{R}^{S}$, that is

[TABLE]

Thus, $\tilde{J}(\theta)=\Phi\theta$. The gradient of $\tilde{J}(i,\theta)$ and the Jacobian of $\tilde{J}(\theta)$ with respect to $\theta$ are

[TABLE]

The goal now is to find a $\tilde{J}$ that is the best approximation of $J$ based on the generated data by applying the stationary policy $\mu=(\mu^{1},\ldots,\mu^{N})$ on the MDP. That is, we seek a weight $\theta^{*}$ such that the distance between $\tilde{J}$ and $J$ is minimized. In our setting, since each agent knows only its own reward function, the agents have to cooperate to find $\theta^{*}$. In the next section, we solve this problem using a distributed variant of the TD$(\lambda)$ algorithm [9], where the agents only share their estimates of the weight $\theta^{*}$ to their neighbors but not their local rewards.

2.1 Distributed TD$(\lambda)$ methods

We introduce a consensus-based variant of the centralized TD$(\lambda)$ method to find the weight $\theta^{*}$; the method is stated formally in Algorithm 1. Our algorithm is a distributed variant of stochastic approximation for solving a suitably reformulated Bellman equation for (1) [1, 2]. Algorithm 1 can be explained as follows.

Each agent $v$ maintains its own estimate $\theta_{v}\in\mathbb{R}^{L}$ of the weight $\theta^{*}$. At every iteration $k\geq 0$, each agent $v$ first performs a weighted average to combine its own estimate and the ones received from its neighbors $u\in{\cal N}_{u}:=\{u\in{\cal V}\;|\;(v,u)\in{\cal E}\}$,

[TABLE]

Agent $v$ then computes the so-called local temporal difference $d_{k}^{v}$ based on the observed data $(s_{k},s_{k+1},r_{k}^{v})$ returned by the environment,

[TABLE]

Here, $d_{k}^{v}$ represents the difference between the local outcome at agent $v$, $r_{k}^{v}+\gamma\tilde{J}(s_{k+1},\theta_{k}^{v})$ and the current estimate $\tilde{J}(s_{k},\theta_{k}^{v})$. Using $y_{k}^{v}$ and $d_{k}^{v}$ agent $v$ then updates its estimate $\theta^{v}$ as

[TABLE]

where $\alpha_{k}$ is a non-negative step size and $z^{v}_{k}$ is the so-called eligibility (or trace) vector. The vector $z^{v}_{k}$ is a weighted combination of all observed feature vectors up to time $k$,

[TABLE]

In addition, the local temporal differences provide the agents an indicator for increasing or decreasing their current state values (by adjusting $\theta^{v}$) after each transition. Finally, each node $v$ returns a time-weighted average $\hat{\theta}_{k}^{v}$

[TABLE]

The updates in Eq. (6) have a simple interpretation: agent $v$ first computes $y_{v}$ by forming a weighted average of its own value $\theta_{v}$ and the values $\theta_{u}$ received from its neighbor $u\in{\cal N}_{v}$, with the goal of seeking consensus on their estimates. Agent $v$ then moves along its own temporal direction $d_{k}^{v}z_{k}^{v}$ to update its estimate, pushing the consensus point toward $\theta^{*}$. In Eq. (6) each agent $v$ only shares $\theta^{v}$ with its neighbors but not its immediate reward $r^{v}$. In a sense, the agents implement in parallel $N$ local TD$(\lambda)$ methods and then combine their estimates through consensus steps to find the global approximate reward $\tilde{J}$.

2.2 Characterization of $\theta^{*}$

The convergence of Algorithm 1 to a point $\theta^{*}$ for the case of single agent was established for all $0\leq\lambda\leq 1$ in [13]. We review below some of the properties of $\theta^{*}$. Let $\pi=(\pi(1),\ldots,\pi(S))$ be the stationary distribution associated with the transition matrix ${\bf P}$, which we assume exists and is unique, and ${\bf D}\in\mathbb{R}^{S\times S}$ be the diagonal matrix whose diagonal entries are $\pi(i)$ for $i\in{\cal S}$. Let $\Pi J$ be the projection of a vector $J$ to the linear subspace spanned by the basis vectors $\phi_{\ell}$,

[TABLE]

where $\|J\|_{{\bf D}}^{2}=J^{T}{\bf D}J$ is the weighted norm of $J$ associated with ${\bf D}$. Moreover, let ${\bf U}\in\mathbb{R}^{S\times S}$ and $b^{v}\in\mathbb{R}^{L}$ for all $v\in{\cal V}$ be defined as

[TABLE]

where $r^{v}\in\mathbb{R}^{S}$ is a vector whose $i$-th component is $r^{v}(i)=\sum_{j=1}^{n}p_{ij}{\cal R}^{v}(i,j)$ for all $v\in{\cal V}$. Then, by [13] we have that the weight $\theta^{*}$ satisfies

[TABLE]

where ${\bf A}$ is a negative definite matrix , i.e., $x^{T}{\bf A}x<0$ $\forall x\in\mathbb{R}^{L}$, satisfying

[TABLE]

In addition, $\theta^{*}$ also satisfies

[TABLE]

As can be seen from (9), when $\lambda=1$ we obtain the best approximation value, i.e., $\Phi\theta^{*}=\Pi J$. However, in the next section we show that although $\lambda<1$ gives a suboptimal solution it converges faster when the noise due to sampling in our algorithm has large variance.

3 Finite-time performance of distributed TD$(\lambda)$

In this section, our goal is to provide a finite-time analysis for the convergence of the distributed TD$(\lambda)$ presented in Algorithm 1. Motivated by the recent work [28] for the single agent problem, we provide an explicit formula for the upper bound on the rates of distributed TD$(\lambda)$ for both constant and time-varying step sizes. The key idea in our analysis is to use the geometric mixing time $\tau$ of the underlying Markov chain; although the “noise” our algorithm encounters is Markovian, its dependence is very weak for samples at least $\tau$ steps apart. A careful characterization of the coupling between these iterates allows us to derive an explicit formula for the convergence rate of distributed TD$(\lambda)$.

3.1 Notation and Assumptions

Let $X_{k}=(s_{k},s_{k+1},z_{k})$ be the Markov chain defined by Algorithm 1, where the discrete-valued $s_{k}\in{\cal S}$ have transition probabilities ${\bf P}$ and the continuous-valued $z_{k}^{v}$ obeys (5). While the $z_{k}^{v}$ are computed locally at each agent, they only depend on the global state sequence, and so will be identical for all $v$; below we will use $z_{k}:=z_{k}^{v}$. Let ${\bf A}(X_{k})\in\mathbb{R}^{L\times L}$ and $b^{v}(X_{k})\in\mathbb{R}^{L}$ be defined as

[TABLE]

Then the update of $\theta_{k}^{v}$ in (6) can be rewritten as

[TABLE]

Let ${\bar{\theta}}$ be the average of $\theta^{v}$, ${\bar{\theta}}=(1/N)\sum_{v}\theta^{v}$. Since ${\bf W}$ is doubly stochastic, taking the average of Eq. (11) yields

[TABLE]

where ${\bar{b}}(X_{k})$ is

[TABLE]

For convenience, let $\Theta$ and ${\bf B}(X_{k})$ be two matrices defined as

[TABLE]

Thus, the matrix form of Eq. (11) can be given as

[TABLE]

Next, we make the following assumptions that are fairly standard in the existing literature of consensus and reinforcement learning [13, 28, 44].

Assumption 1**.**

The matrix ${\bf W}$, whose $(i,j)$-th entries are $w_{uv}$, is doubly stochastic, i.e., $\sum_{u=1}^{n}w_{uv}=1$ for all $j$ and $\sum_{v=1}^{n}w_{uv}=1$ for all $i$. Finally, $w_{uu}>0$ and the weights $w_{uv}>0$ if and only if $(u,v)\in{\cal E}$ otherwise $w_{uv}=0$.

Assumption 2**.**

All the local rewards are uniformly bounded, i.e., there exist a constant $R$, for all $v\in{\cal V}$ such that $|\,{\cal R}^{v}(i,j)\,|\leq R$, for all $i,j\in{\cal S}$.

Assumption 3**.**

The feature vectors $\{\phi_{\ell}\}$, for all $\ell\in\{1,\ldots,L\}$, are linearly independent, i.e., the matrix $\Phi$ has full column rank. In addition, we assume that all feature vectors $\phi(s)$ are uniformly bounded, i.e., $\|\phi(s)\|\leq 1$.

Assumption 4**.**

The Markov chain associated with ${\bf P}$ is irreducible and aperiodic.

Assumption 1 implies that ${\bf W}$ has a largest singular value of $1$, and its other singular values are strictly less than $1$; see for example, the Perron-Frobenius theorem [45]. We denote by $\sigma_{2}\in(0,1)$ the second largest singular value of ${\bf W}$, which is a key quantity in the analysis of the mixing time of a Markov chain with transition probabilities given by ${\bf W}$.

Under Assumption 2 the accumulative reward $J^{*}$ is well defined, while under Assumption 3, the projection operator $\Pi$ is well defined. If there are some dependent $\phi_{\ell}$, we can simply disregard those dependent feature vectors. Note that the uniform boundedness of $\phi_{\ell}$ can be guaranteed through feature normalization. Moreover, Assumption 4 implies that the underlying Markov chain ${\bf P}$ is ergodic. First, this guarantees that there exists a unique stationary distribution $\pi$ with positive entries and

[TABLE]

Second, it implies that the Markov chain mixes at a geometric rate [46]. In particular, we consider the following definition of the mixing time of a Markov chain.

Definition 1**.**

Given a positive constant $\alpha$, we denote by $\tau(\alpha)$ the mixing time of the Markov chain $\{X_{k}\}$ given as

[TABLE]

where $b$ and ${\bf A}$ are given in Eqs. (7) and (8), respectively. In addition, under Assumption 4 the Markov chain $\{X_{k}\}$ has a geometric mixing time [46], i.e., there exist a constant $C$ such that given a small constant $\alpha$ we have

[TABLE]

Finally, using Eq. (10) and Assumptions 2 and 3 the induced $2$-norm of $\|{\bf A}(X_{k})\|$ and $\|b^{v}(X_{k})\|$ can be upper bounded by

[TABLE]

In addition, since $\lim_{k\rightarrow\infty}\mathbb{E}[{\bf A}(X_{k})]={\bf A}$ and $\lim_{k\rightarrow\infty}\mathbb{E}[{\bar{b}}(X_{k})]=b$ we also have

[TABLE]

where recall that ${\bf A}$ and $b$ are given in Eq. (7).

3.2 Main Results

In this section, we provide the main results of this paper, which are the finite-time analysis for the convergence of distributed TD$(\lambda)$. We first denote by $\delta$ a positive constant satisfying

[TABLE]

where $\sigma_{2}\in(0,1)$ is the second largest singular value of ${\bf W}$ and $\alpha$ is a constant chosen such that $\delta\in(0,1)$. Our first result is to establish the rate of convergence of the distributed TD$(\lambda)$ when the step size is this constant $\alpha$. In particular, under some proper choice of $\alpha$, we show that the average mean-squared error across the nodes converges with a linear rate to a ball centered at the $\theta^{*}$. In addition, the size of this ball is a function of $\alpha$. The following theorem states this result formally.

Theorem 1**.**

Suppose that Assumptions 1–4 hold. Let $\{\theta_{k}^{v}\}$, for all $v\in{\cal V}$, be generated by Algorithm 1 and $\sigma_{\min}>0$ be the smallest singular value of $-{\bf A}$. Let $\Psi_{1}$ and $\Psi_{2}$ be the constants defined

[TABLE]

Let $\alpha_{k}=\alpha$ and $\tau(\alpha)$ be the corresponding mixing time defined in (22), where $\alpha$ satisfies

[TABLE]

Then we have for all $k\geq\tau(\alpha)$

[TABLE]

where $C$ is given in (23) and $\delta$ is defined in (26).

Remark 1*.*

We first note that by (23) we have

[TABLE]

Thus, one can choose such an $\alpha$ to satisfy Eq. (28). Second, as can be seen from our result in (29) that the variance in our algorithm, i.e., the quantity

[TABLE]

theoretically explains the empirical observation of TD$(\lambda)$, where $\lambda=1$ gives the best approximation and $\lambda={0}$ yields faster convergence under large variance. Indeed, the quantity above is smallest when $\lambda={0}$ and gets larger as $\lambda$ increases to $1$. On the other hand, when $\lambda=1$ one can see from (9) that $\tilde{J}=\pi J$, the best approximation of $J$ in the subspace spanned by the feature vectors.

Our second main result is to show the rate of distributed TD$(\lambda)$ under time-varying step sizes $\alpha_{k}=\alpha_{0}\,/\,(k+1)$ for some positive constant $\alpha_{0}$. Under this condition, we show that the average of mean square errors at each agent asymptotically converges to [math] with a sublinear rate, which is formally stated in the following theorem.

Theorem 2**.**

Suppose that Assumptions 1–4 hold. Let the sequence $\{\theta_{k}^{v}\}$, for all $v\in{\cal V}$, be generated by Algorithm 1. Let $\sigma_{\min}>0$ be the smallest singular value of $-{\bf A}$ and $\alpha_{k}=\alpha_{0}\,/\,(k+1)$ for some $\alpha_{0}\geq 1/\sigma_{\min}$. Denote by $\Psi_{3}$ and $\Psi_{4}$ two constants as

[TABLE]

In addition, let ${\cal K}^{*}$ be a positive integer such that for all $k\geq{\cal K}^{*}$

[TABLE]

where $\alpha$ satisfies (28) and $\delta$ is defined in (26). Then we have for all $k\geq{\cal K}^{*}$

[TABLE]

4 Finite-time analysis of distributed TD$(\lambda)$

In this section, we provide the analysis for the main results in this paper, that is, the proofs of Theorems 1 and 2. We start by first considering an upper bound for the consensus error, $\|\Theta-{\mathbf{1}}{\bar{\theta}}^{T}\|$, which measures the difference between the agents’ estimate $\theta_{v}$ and their average ${\bar{\theta}}$. The results in this lemma will be used in the analysis of both Theorems 1 and 2, so we present here for convenience.

Lemma 1**.**

Suppose that Assumptions 1–3 hold. Let the sequence $\{\theta_{k}^{v}\}$, for all $v\in{\cal V}$, be generated by Algorithm 1. We consider the following two cases

Let $\sigma_{2}\in(0,1)$ be the second largest singular value of ${\bf W}$ and $\alpha_{k}=\alpha$ satisfy

[TABLE]

In addition, let $\delta\in(0,1)$ be defined in (26). Then we have for all $k\geq 0$

[TABLE] 2. 2.

Let $\alpha_{k}=\alpha_{0}/(k+1)$ for some positive constant $\alpha_{0}$ and ${\cal K}^{*}_{1}$ be a positive constant such that

[TABLE]

where $\alpha$ satisfies Eq. (33). Then we obtain for all $k\geq{\cal K}^{*}$ that

[TABLE]

4.1 Constant Step Sizes

We now consider the case where step sizes $\alpha_{k}$ are constants, i.e.. $\alpha_{k}=\alpha$, for some positive constant $\alpha$. To show the result in Theorem 1, we first consider the following lemma about an useful error bound of the bias term generated by Algorithm 1. For an ease of exposition, the proof of this lemma is presented in the Appendix B.

Lemma 2**.**

Suppose that Assumptions 1–4 hold. Let the sequence $\{\theta_{k}^{v}\}$, for all $v\in{\cal V}$, be generated by Algorithm 1. Let $\alpha_{k}=\alpha$ satisfying (28). Then for all $k\geq\tau(\alpha)$ we have

[TABLE]

We are now ready to proceed the analysis of Theorem 1 as follows.

Proof of Theorem 1.

Recall from Eq. (12) with $\alpha_{k}=\alpha$ that

[TABLE]

which gives for all $k\geq 0$

[TABLE]

where in the last inequality we use Eq. (24) and the fact that ${\bf A}\theta^{*}=-b$. Next, since ${\bf A}$ is negative definite and let $\sigma_{\min}>0$ be the smallest singular value of $-{\bf A}$, we have

[TABLE]

We now take the expectation on both sides of Eq. (37) and use Eqs. (36) and (38) to have

[TABLE]

which by applying the Cauchy-Schwarz inequality to the second term yields

[TABLE]

where $\Psi_{1}$ and $\Psi_{2}$ are two constants defined as

[TABLE]

Since $\alpha$ satisfies Eq. (28) we have $-2\sigma_{min}+\Psi_{1}\alpha\leq-\sigma_{min}$, which by substituting into Eq. (39) yields for all $k\geq\tau(\alpha)$

[TABLE]

On the other hand, using Eq. (12) with $\alpha_{k}=\alpha$ we consider

[TABLE]

which by using Eq. (24) yields

[TABLE]

where the last inequality is due to the relation $1+x\leq\exp(x)$ for any $x\geq 0$. Thus, since $\alpha$ satisfies (28), i.e., $\frac{\alpha\tau(\alpha)(1+\gamma)}{1-\gamma\lambda}\leq\log(2)$, the preceding relation yields

[TABLE]

which implies that

[TABLE]

Substituting the preceding relation into Eq. (40) yields

[TABLE]

Next, note that $\alpha$ also satisfies the condition in Eq. (33), hence, by Eq. (34) we have

[TABLE]

which by using Eq. (41) gives Eq. (29), i.e.,

[TABLE]

∎

4.2 Time-Varying Step Sizes

In this section, we provide the analysis of Theorem 2, that is, we consider the case $\alpha_{t}=\alpha_{0}\,/\,(t+1)$ for some positive constant $\alpha_{0}$. Similar to the case of constant step sizes, we consider the following important lemma about the bias term in our analysis considered later. The analysis of this lemma is presented in Appendix for completeness.

Lemma 3**.**

Suppose that Assumptions 1–4 hold. Let the sequence $\{\theta_{k}^{v}\}$, for all $v\in{\cal V}$, be generated by Algorithm 1. Let $\alpha_{k}$ be a nonnegative and nonincreasing sequence of step sizes satisfying $\lim_{k\rightarrow\infty}\alpha_{k}=0$, and let $\tau(\alpha_{k})$ be the mixing time associated with $\alpha_{k}$. We denote by ${\cal K}^{*}$ a positive integer defined in (31). Then for all $k\geq{\cal K}^{*}$ we have

[TABLE]

Using the preceding lemma, we now show the result stated in Theorem 2. The analysis is similar to the one of Theorem 1.

Proof of Theorem 2.

Similar to Eq. (39), by using Eq. (42) we obtain for all $k\geq{\cal K}^{*}$

[TABLE]

which yields

[TABLE]

where $\Psi_{3}$ and $\Psi_{4}$ are two constants defined as

[TABLE]

Recall from (31) that

[TABLE]

which by substituting into the preceding relation yields for all $k\geq{\cal K}^{*}$

[TABLE]

where in the last inequality we use $\alpha_{0}\geq 1/\sigma_{\min}$ and Eq. (23). Iteratively updating Eq. (44) we have for all $k\geq{\cal K}^{*}$

[TABLE]

where the third inequality we use

[TABLE]

and the last inequality is due to the integral test

[TABLE]

Next, by Eq. (35) we have for all $k\geq{\cal K}^{*}$

[TABLE]

which by using Eq. (45) gives Eq. (32), i.e., for all $k\geq{\cal K}^{*}$

[TABLE]

∎

5 Conclusion and Discussion

In this paper, we consider a distributed consensus-based variant of the popular TD$(\lambda)$ algorithm for estimating the value function of a given stationary policy. Our main contribution is to provide a finite-time analysis for the performance of distributed TD$(\lambda)$, which has not been addressed in the existing literature of MARL. Our results theoretically address some numerical observations of TD$(\lambda)$ , that is, $\lambda=1$ gives the best approximation of the function values while $\lambda=0$ leads to better performance when there is large variance in the algorithm. One interesting question left from this work is the finite-time analysis when the policy is not stationary, e.g., distributed actor-critic methods. We believe that this paper establishes fundamental results that enable one to tackle these problems, which we leave for our future research.

Appendix A Proof of Lemma 1

For convenience, we consider the following notation

[TABLE]

By Assumption 1, ${\bf W}$ is doubly stochastic, which yields

[TABLE]

Thus, using Eqs. (12) and (21) we have

[TABLE]

which by applying the $2$-norm on both sides and using the triangle inequality yields

[TABLE]

First, since ${\bf W}$ satisfies Assumption 1, by the Courant-Fisher theorem [45] we have

[TABLE]

Second, using the definition of ${\bf B}(X_{k})$ in (20) and the relation in Eq. (24) we obtain

[TABLE]

Substituting the previous two relations into Eq. (46) we have

[TABLE]

We now consider the following two cases for different choices of the step sizes $\alpha_{k}$.

Let $\alpha_{k}=\alpha$ for some constant $\alpha$ satisfying Eq. (33), which gives

[TABLE]

Thus, using this relation into Eq. (47) gives Eq. (34), i.e.,

[TABLE]

where in the last inequality we use $\|{\bf Y}_{0}\|=\|{\bf Q}\Theta_{0}\|\leq\|{\bf Q}\|\|\Theta_{0}\|\leq\|\Theta_{0}\|.$ 2. 2.

Let $\alpha_{k}$ be a sequence of step sizes satisfying $\alpha_{k}=\alpha_{0}/(k+1)$ for some constant $\alpha_{0}$. In addition, we have

[TABLE]

Next, using Eq. (47) above, we have for all $k\geq{\cal K}^{*}$ that

[TABLE]

Appendix B Constant step sizes

In this section, we provide a full analysis of Lemma 2. To to that, we first study a few upper bounds of the quantity $\|{\bar{\theta}}_{t}-{\bar{\theta}}_{t-\tau(\alpha)}\|^{2}$, which is the change of ${\bar{\theta}}_{t}$ in any $\tau(\alpha)$ mixing time interval. Our main observation states that under some proper choice of the step size $\alpha$, such difference should not be too large, as stated in the following lemma. In addition, these bounds will play a crucial role for our result given later.

Lemma 4**.**

Suppose that Assumptions 1–3 hold. Let the sequence $\{\theta_{k}^{v}\}$, for all $v\in{\cal V}$, be generated by Algorithm 1. In addition, let $\alpha_{k}=\alpha$ and $\tau(\alpha)$ be the corresponding mixing time in (22) satisfying

[TABLE]

Then for all $k\geq\tau(\alpha)$ we have

[TABLE]

Proof.

Using Eq. (12) with $\alpha_{k}=\alpha$ we have

[TABLE]

which when taking the $2$-norm on both sides and using Eq. (24) yields

[TABLE]

First, we consider when $k\geq\tau(\alpha)$, where $\tau(\alpha)$ is the mixing time associated with $\alpha$ defined in (22). Indeed, using Eq. (53) we have for all $u\in[k-\tau(\alpha),k]$ and $k\geq\tau(\alpha)$

[TABLE]

Recall from Eq. (49) that $\alpha$ satisfies

[TABLE]

which by using the relation $1+x\leq e^{x}$ for all $x\geq 0$ we have for all $u\in[k-\tau(\alpha),k]$

[TABLE]

Using the preceding relation into Eq. (54) we have for all $u\in[k-\tau(\alpha),k]$ for $u\geq\tau(\alpha)$

[TABLE]

Thus, using Eq. (56) we obtain Eq. (50) for all $k\geq\tau(\alpha)$ , i.e.,

[TABLE]

where in the last inequality we use Eq. (49), i.e.,

[TABLE]

Next, using the equation above we have for all $k\geq\tau(\alpha)$

[TABLE]

which gives Eq. (51), i.e.,

[TABLE]

Using the inequality $(x+y)^{2}\leq 2x^{2}+2y^{2}$ for all $x,y\in\mathbb{R}$, the preceding equation also gives Eq. (52), i.e.,

[TABLE]

where in the last inequality we use $(1+\gamma)\alpha\tau(\alpha)\leq(1-\gamma\lambda)\log(2).$ ∎

Next, using the previous results we now consider the following two important lemmas. We denote by ${\cal F}_{k}=\{\theta_{0},\ldots,\theta_{k}\}$ the filtration containing all the information generated by Algorithm 1 up to time $k$.

Lemma 5**.**

Suppose that Assumptions 1–4 hold. Let the sequence $\{\theta_{k}^{v}\}$, for all $v\in{\cal V}$, be generated by Algorithm 1. In addition, let $\alpha_{t}=\alpha$ where $\alpha$ satisfies (49). Then for all $k\geq\tau(\alpha)$ we have

[TABLE]

Proof.

First, using the mixing time defined in (22) we consider for all $k\geq\tau(\alpha)$

[TABLE]

which by using Eq. (52) yields Eq.  (57), i.e.,

[TABLE]

Next, we use the definition of mixing time in Eq. (22) again to have

[TABLE]

which by using Eq. (52) yields Eq. (58), i.e.,

[TABLE]

∎

Lemma 6**.**

Suppose that Assumptions 1–4 hold. Let the sequence $\{\theta_{k}^{v}\}$, for all $v\in{\cal V}$, be generated by Algorithm 1. In addition, let $\alpha_{k}=\alpha$ where $\alpha$ satisfies (49). Then for all $k\geq\tau(\alpha)$ we have

[TABLE]

Proof.

First, we show Eq. (59). using Eqs. (24) and (25) we have for all $k\geq\tau(\alpha)$

[TABLE]

which by using Eq. (50) to upper bound the term $\|{\bar{\theta}}_{k}-{\bar{\theta}}_{k-\tau(\alpha)}\|$ yields

[TABLE]

Using the relation $2xy\leq x^{2}+y^{2}$ to the second term on the right-hand side of above yields

[TABLE]

which by using Eq. (52) to the first term on the right-hand side we obtain

[TABLE]

Next, using Eqs. (24) and (25) again we have

[TABLE]

which by using (50) and $2x\leq x^{2}+1$ yields

[TABLE]

Using Eq. (52) to the preceding equation we have

[TABLE]

Third, using the first inequality in (52) yields

[TABLE]

where in the last inequality we use Eq. (49) to have $(1+\gamma)\alpha\tau(\alpha)/(1-\gamma\lambda)\leq\log(2)\leq 1/3.$ Finally, using Eqs. (24) and (25) we get Eq. (62), i.e.,

[TABLE]

where the last inequality we use $2x\leq x^{2}+1$. ∎

Finally, using Lemmas 4–6 we now ready to show Lemma 2.

Proof of Lemma 2.

First we consider

[TABLE]

which by using Eqs. (57) and (59)–(61) we have

[TABLE]

Similarly, we consider

[TABLE]

which by using Eqs. (58) and (62) yields

[TABLE]

Using Eqs. (64) and (65) we obtain

[TABLE]

∎

Appendix C Time-varying step sizes

Similarly, to show Lemma 3 we first consider the following sequence of lemmas.

Lemma 7**.**

Suppose that Assumptions 1–4 hold. Let the sequence $\{\theta_{k}^{v}\}$, for all $v\in{\cal V}$, be generated by Algorithm 1 and let $\alpha_{k}$ satisfy the conditions in Lemma 3. Denote by $\alpha_{k,\tau(\alpha_{k})}$

[TABLE]

Then for all $k\geq{\cal K}^{*}$ we have

[TABLE]

Proof.

Let $\tau(\alpha_{k})$ be the mixing time associated with $\alpha_{k}$ defined in (22). First, using Eq. (12) we have

[TABLE]

which when taking the $2$-norm on both sides and using Eq. (24) yields

[TABLE]

Using the relation $1+x\leq\exp(x)$ for all $x\geq 0$ we have from the preceding relation for some $t\in[k-\tau(\alpha_{k}),\;k-1]$

[TABLE]

Recall from (31) that ${\cal K}^{*}$ is a positive integer such that

[TABLE]

which by using the equation above we have for all $t\in[k-\tau(\alpha_{k}),\;k-1]$ with $k\geq{\cal K}^{*}$

[TABLE]

Thus, using the preceding relation yields Eq. (68), i.e., for all $k\geq{\cal K}^{*}$

[TABLE]

where the last inequality we use

[TABLE]

Next, by triangle inequality we have

[TABLE]

which by using Eq. (71) gives Eq. (69), i.e., for all $k\geq{\cal K}^{*}$

[TABLE]

Finally, using the inequality $(a+b)^{2}\leq 2a^{2}+2b^{2}$ and Eq. (71) we obtain Eq. (70), i.e., for all $k\geq{\cal K}^{*}$

[TABLE]

∎

Lemma 8**.**

Suppose that Assumptions 1–4 hold. Let the sequence $\{\theta_{k}^{v}\}$, for all $v\in{\cal V}$, be generated by Algorithm 1. Let $\alpha_{k}$ satisfy all the conditions in Lemma 7. Then for all $k\geq{\cal K}^{*}$ we have

[TABLE]

Proof.

Recall that ${\cal F}_{k}$ is the filtration containing all the history up to time $k$. Let $\tau(\alpha_{k})$ be the mixing time associated with step size $\alpha_{k}$ defined in (22). Then we first have $k\geq\tau(\alpha_{k})$

[TABLE]

which by using Eq. (70) yields Eq.  (72), i.e.,

[TABLE]

Next, we use the definition of mixing time in Eq. (22) again to have

[TABLE]

which by using Eq. (70) yields Eq. (73), i.e.,

[TABLE]

∎

Lemma 9**.**

Suppose that Assumptions 1–4 hold. Let the sequence $\{\theta_{k}^{v}\}$, for all $v\in{\cal V}$, be generated by Algorithm 1. Let $\alpha_{k}$ satisfy all the conditions in Lemma 7. Then for all $k\geq{\cal K}^{*}$ we have

[TABLE]

Proof.

First, we show Eq. (74). Using Eqs. (24) and (25) we have for all $k\geq\tau(\alpha_{k})$

[TABLE]

which by using Eq. (68) to upper bound the term $\|{\bar{\theta}}_{k}-{\bar{\theta}}_{k-\tau(\alpha_{k})}\|$ yields

[TABLE]

Using the relation $2xy\leq x^{2}+y^{2}$ to bound the second and third terms on the right-hand side of the preceding relation yields

[TABLE]

which by using Eq. (70) to the first term on the right-hand side we obtain Eq. (74), i.e.,

[TABLE]

Next, using Eqs. (24) and (25) again we have

[TABLE]

which by using Eq. (68) and $2x\leq x^{2}+1$ yields

[TABLE]

Using Eq. (70) to bound the preceding relation gives Eq. (75), i.e.,

[TABLE]

Third, using the first inequality in (70) yields Eq. (76), i.e.,

[TABLE]

where in the last inequality we use Eq. (31) to have $(1+\gamma)\alpha_{k,\tau(\alpha_{k})}/(1-\gamma\lambda)\leq\log(2)\leq\frac{1}{3},\;\forall k\geq{\cal K}^{*}.$ Finally, using Eqs. (24) and (25) we get Eq. (77), i.e.,

[TABLE]

where in the last inequality we use $2x\leq x^{2}+1$. ∎

Finally, using Lemmas 7–9 we now show Lemma 3.

Proof of Lemma 3.

First we consider

[TABLE]

which by using Eqs. (72) and (74)–(76) we have

[TABLE]

where in the last inequality we use $\alpha_{k}\leq\alpha_{k-\tau(\alpha_{k})}$ and $\alpha_{k,\tau(\alpha_{k})}\leq\tau(\alpha_{k})\alpha_{k-\tau(\alpha_{k})}$. Similarly, we consider

[TABLE]

which by using Eqs. (73) and (77) yields

[TABLE]

Using Eqs. (78) and (79) we obtain Eq. (42), i.e.,

[TABLE]

∎