A Convergent Online Single Time Scale Actor Critic Algorithm

TL;DR

This paper introduces a novel online actor-critic reinforcement learning algorithm where both actor and critic operate on the same time scale, simplifying convergence analysis and potentially enhancing biological plausibility.

Contribution

It presents a convergent single time scale actor-critic algorithm using temporal difference learning, unlike previous methods requiring different time scales for convergence.

Findings

Converges to a neighborhood of a local maximum of average reward.

Uses the same temporal difference signal for both actor and critic.

Operates with both components on a similar time scale.

Abstract

Actor-Critic based approaches were among the first to address reinforcement learning in a general setting. Recently, these algorithms have gained renewed interest due to their generality, good convergence properties, and possible biological relevance. In this paper, we introduce an online temporal difference based actor-critic algorithm which is proved to converge to a neighborhood of a local maximum of the average reward. Linear function approximation is used by the critic in order estimate the value function, and the temporal difference signal, which is passed from the critic to the actor. The main distinguishing feature of the present convergence proof is that both the actor and the critic operate on a similar time scale, while in most current convergence proofs they are required to have very different time scales in order to converge. Moreover, the same temporal difference signal is used to update the parameters of both the actor and the critic. A limitation of the proposed approach, compared to results available for two time scale convergence, is that convergence is guaranteed only to a neighborhood of an optimal value, rather to an optimal value itself. The single time scale and identical temporal difference signal used by the actor and the critic, may provide a step towards constructing more biologically realistic models of reinforcement learning in the brain.

Figures

Equations

$$P(y|x,\theta )={\int }_{\mathcal{U}}P(y|x,u)d\mu (u|x,\theta ).$$

$$\left|\frac{\partial \mu (u|x,\theta )}{\partial {\theta }_k}\right|\le B_{{\mu }_1},\left|\frac{{\partial }^2\mu (u|x,\theta )}{\partial {\theta }_{k_1}\partial {\theta }_{k_2}}\right|\le B_{{\mu }_2}.$$

$$J(x,\theta )\triangleq \underset{T\to \infty }{\lim }\text{E}\left[\frac{1}{T}\sum _{n=0}^{T-1}r(x_n)|x_0=x,\theta \right],$$

$$T\triangleq \min \{k>0|x_0=x^{\ast },x_k=x^{\ast }\}.$$

$$\eta (\theta )=\underset{T\to \infty }{\lim }\text{E}\left[\frac{1}{T}\sum _{n=0}^{T-1}r(x_n)|x_0=x^{\ast },\theta \right].$$

$$h(x,\theta )\triangleq \text{E}\left[\sum _{n=0}^{T-1}(r(x_n)-\eta (\theta ))|x_0=x,\theta \right].$$

$$h(x,\theta )=r(x)-\eta (\theta )+\sum _{y\in \mathcal{X}}P(y|x,\theta )h(y,\theta ).$$

$$d(x,y,\theta )\triangleq r(x)-\eta (\theta )+h(y,\theta )-h(x,\theta ).$$

$$\left|h(x,\theta )\right|\le B_h,\left|\frac{\partial h(x,\theta )}{\partial {\theta }_i}\right|\le B_{h_1},\left|\frac{{\partial }^{2}h(x,\theta )}{\partial {\theta }_i\partial {\theta }_j}\right|\le B_{h_2}.$$

$$\psi (x,u,\theta )\triangleq \frac{{\nabla }_{\theta }\mu (u|x,\theta )}{\mu (u|x,\theta )},$$

$${\Vert \psi (x,u,\theta )\Vert }_2\le B_{\psi }<\infty ,$$

$${\nabla }_{\theta }\eta (\theta )=\sum _{x,y\in \mathcal{X}}P(x,u,y,\theta )\psi (x,u,\theta )d(x,y,\theta ),$$

$$I(w,\theta )\triangleq \frac{1}{2}\sum _{x\in \mathcal{X}}\pi (x|\theta ){\left(\tilde{h}(x,w)-h(x,\theta )\right)}^2\text{leavevmode}.$$

$$\tilde{h}(x,w)=\varphi (x{)}^{\prime }w,$$

$$\Phi\triangleq\left(\begin{array}[]{cccc}\phi_{1}(x_{1})&\phi_{2}(x_{1})&\ldots&\phi_{L}(x_{1})\\ \phi_{1}(x_{2})&\phi_{2}(x_{2})&\ldots&\phi_{L}(x_{2})\\ \vdots&\vdots&&\vdots\\ \phi_{1}(x_{|\mathcal{X}|})&\phi_{2}(x_{|\mathcal{X}|})&\ldots&\phi_{L}(x_{|\mathcal{X}|})\end{array}\right)\leavevmode\nobreak\ ,$$

$$\tilde{h}(w)=\Phi w,$$

$$I(w,\theta )=\frac{1}{2}{\Vert \Pi (\theta {)}^{\frac{1}{2}}\left(h(\theta )-\Phi w\right)\Vert }_2^2\triangleq \frac{1}{2}{\Vert h(\theta )-\Phi w\Vert }_{\Pi (\theta )}^2.$$

$$w_{n+1}=w_n-{\gamma }_n{\nabla }_{w}I(w_n,\theta ),$$

$$\sum _{n=1}^{\infty }{\gamma }_n=\infty ,\sum _{n=1}^{\infty }{\gamma }_n^2<\infty .$$

$${\nabla }_{w}I(w_n)={\Phi }^{\prime }\Pi (\theta )\Phi w_n-{\Phi }^{\prime }\Pi (\theta )h(\theta ).$$

$$w^{\ast }(\theta )={\left({\Phi }^{\prime }\Pi (\theta )\Phi \right)}^{-1}{\Phi }^{\prime }\Pi (\theta )h(\theta ).$$

$$\begin{array}{rl}{w}_{n+1}& =w_n-{\gamma }_n{\Phi }^{\prime }\Pi \left(\theta \right)\left(\Phi w_n-h(\theta )\right).\end{array}$$

$${ϵ}_{\text{app}}\left(\theta \right)\triangleq \underset{w\in {\mathbb{R}}^L}{\inf }{\Vert h\left(\theta \right)-\Phi w\Vert }_{\pi \left(\theta \right)}={\Vert h\left(\theta \right)-\Phi w^{\ast }\left(\theta \right)\Vert }_{\pi \left(\theta \right)}.$$

$${ϵ}_{\text{app}}\triangleq \underset{\theta \in {\mathbb{R}}^K}{\sup }{ϵ}_{\text{app}}\left(\theta \right).$$

$$w_{n+1}=w_n+{\gamma }_n{\Phi }^{\prime }\Pi (\theta )\left({\hat{h}}_n-\Phi w_n\right).$$

$${\hat{h}}_{n+1}\left(x_n\right)$$

$${\hat{h}}_{n+1}^{\left(k\right)}\left(x_n\right)=\left(\sum _{m=0}^{k}r\left(x_{n+m}\right)+{\hat{h}}_n\left(x_{n+k+1}\right)\right).$$

$$w_{n+1}$$

$$e_n$$

$$w_{n+1}$$

$$\tilde{d}\left(x_n,x_{n+1},w_n\right)$$

Full text

1 Introduction

In Reinforcement Learning (RL) an agent attempts to improve its performance over time at a given task, based on continual interaction with the (usually unknown) environment (Bertsekas and Tsitsiklis (1996); Sutton and Barto (1998)). Formally, it is the problem of mapping situations to actions in order to maximize a given average reward signal. The interaction between the agent and the environment is modeled mathematically as a Markov Decision Process (MDP). Approaches based on a direct interaction with the environment, are referred to as simulation based algorithms, and will form the major focus of this paper.

A well known subclass of RL approaches consists of the so called actor-critic (AC) algorithms (e.g., Sutton and Barto (1998)), where the agent is divided into two components, an actor and a critic. The critic functions as a value estimator, whereas the actor attempts to select actions based on the value estimated by the critic. These two components solve their own problems separately but interactively. Many methods for solving the critic’s value estimation problem, for a fixed policy, have been proposed, but, arguably, the most widely used is temporal difference (TD) learning. TD learning was demonstrated to accelerate convergence by trading bias for variance effectively Singh and Dayan (1998), and is often used as a component of AC algorithms.

In general, policy selection may be randomized. When facing problems with a large number of states or actions (or even continuous state-action problems), effective policy selection may suffer from several problems, such as slow convergence rate or an inefficient representation of the policy. A possible approach to policy learning is the so-called policy gradient method (Baxter and Bartlett (2001); Cao (2007); Cao and Chen (1997); Konda and Tsitsiklis (2003); Marbach and Tsitsiklis (1998)). Instead of maintaining a separate estimate for the value for each state (or state-action pair), the agent maintains a parametrized policy function. The policy function is taken to be a differentiable function of a parameter vector and of the state. Given the performance measure, depending on the agent’s policy parameters, these parameters are updated using a sampling-based estimate of the gradient of the average reward. While such approaches can be proved to converge under certain conditions (e.g., Baxter and Bartlett (2001)), they often lead to slow convergence, due to very high variance. A more general approach based on sensitivity analysis, which includes policy gradient methods as well as non-parametric average reward functions, has been discussed in depth in the recent manuscript by Cao (2007).

Several AC algorithms with associated convergence proofs have been proposed recently (a short review is given in section 2.2). As far as we are aware, all the convergence results for these algorithms are based on two time scales, specifically, the actor is assumed to update its internal parameters on a much slower time scale than the one used by the critic. The intuitive reason for this time scale separation is clear, since the actor improves its policy based on the critic’s estimates. It can be expected that rapid change of the policy parameters may not allow the critic to effectively evaluate the value function, which may lead to instability when used by the actor in order to re-update its parameters.

The objective of this paper is to propose an online AC algorithm and establish its convergence under conditions which do not require the separation into two time scales. There is clear theoretical motivation for such an approach, as it can potentially lead to faster convergence rates, although this is not a an issue we stress in this work. In fact, our motivation for the current direction was based on the possible relevance of AC algorithms in a biological context (e.g, Daw et al. (2006)), where it would be difficult to justify two very different time scales operating within the same anatomical structure. We refer the reader to DiCastro et al. (2008) for some preliminary ideas and references related to these issues. Given the weaker conditions assumed on the time scales, our convergence result is, not surprisingly, somewhat weaker than that provided recently in (e.g., Bhatnagar et al. (2008a, b)), as we are not ensured to converge to a local optimum, but only to a neighborhood of such an optimum. Nevertheless, it is shown that the neighborhood size can be algorithmically controlled. Further comparative discussion can be found in section 2.

This paper is organized as follows. In section 2 we briefly recapitulate current AC algorithms for which convergence proofs are available. In section 3, we formally introduce the problem setup. We begin section 4 by relating the TD signal to the gradient of the average reward, and then move on to motivate and derive the main AC algorithm, concluding the section with a convergence proof. A comparative discussion of the main features of our approach is presented in section 5, followed by some simulation results in section 6. Finally, in section 7, we discuss the results and point out possible future work. In order to facilitate the readability of the paper, we have relegated all technical proofs to appendices.

2 Previous Work

In this section we briefly review some previous work in RL which bears direct relevance to our work. While many AC algorithms have been introduced over the years, we focus only on those for which a convergence proof is available, since the main focus of this work is on convergence issues, rather than on establishing the most practically effective algorithms (see, for example, Peters and Schaal (2008), for promising applications of AC algorithms in a robotic setting).

2.1 Direct policy gradient algorithms

Direct policy gradient algorithms, employing agents which consist of an actor only, typically estimate a noisy gradient of the average reward, and are relatively close in their characteristics to AC algorithms. The main difference from the latter is that the agent does not maintain a separate value estimator for each state, but rather interacts with the environment directly, and in a sense maintains its value estimate implicitly through a mapping which signifies which path the agent should take in order to maximize its average reward per stage.

Marbach and Tsitsiklis (1998) suggested an algorithm for non-discounted environments. The gradient estimate is based on an estimate of the state values which the actor estimates while interacting with the environment. If the actor returns to a sequence of previously visited states, it re-estimates the states value, not taking into account its previous visits. This approach often results in large estimation variance.

Baxter and Bartlett (2001) proposed an online algorithm for partially observable MDPs. In this algorithm, the agent estimates the expected average reward for the non-discounted problems through an estimate of the value function of a related discounted problem. It was shown that when the discount factor approaches $1$, the related discounted problem approximates the average reward per stage. Similar to the algorithms in (Marbach and Tsitsiklis (1998)), it suffers from relatively large estimation variance. In (Baxter et al. (2004)), a method was proposed for coping with the large variance by adding a baseline to the value function estimation.

2.2 Actor Critic Algorithms

As stated in section 1, the convergence proofs of which we are aware for AC algorithms are based on two time scale stochastic approximation (Borkar (1997)), where the actor is assumed to operate on a time scale which is much slower than that used by the critic.

Konda and Borkar (1999) suggested a set of AC algorithms. In two of their algorithms (Algorithms 3 and 6), parametrized policy based actors were used while the critic was based on a lookup table. Those algorithms and their convergence proofs were specific to the Gibbs policy function in the actor.

As far as we are aware, Konda and Tsitsiklis (2003) provided the first convergence proof for an AC algorithm based on function approximation. The information passed from the critic to the actor is the critic’s action-value function, and the critic’s basis functions, which are explicitly used by the actor. They provided a convergence proof of their TD($\lambda$) algorithm where $\lambda$ approaches $1$. A drawback of the algorithm is that the actor and the critic must share the information regarding the actor’s parameters. This detailed information sharing is a clear handicap in a biological context, which was one of the driving forces for the present work.

Finally, Bhatnagar et al. (2008a, b) recently proposed an AC algorithm which closely resembles our proposed algorithm, and which was developed independently of ours. In this work the actor uses a parametrized policy function while the critic uses a function approximation for the state evaluation. The critic passes to the actor the TD(0) signal and based on it the actor estimates the average reward gradient. A detailed comparison will be provided in section 5. As pointed out in Bhatnagar et al. (2008a, b), their work is the first to provide a convergence proof for an AC algorithm incorporating bootstrapping Sutton and Barto (1998), where bootstrapping refers to a situation where estimates are updated based on other estimates, rather than on direct measurements (as in Monte Carlo approaches). This feature applies to our work as well. We also note that Bhatnagar et al. (2008a, b) extend their approach to the so-called natural gradient estimator, which has been shown to lead to improved convergence in supervised learning as well as RL. The present study focuses on the standard gradient estimate, leaving the extension to natural gradients to future work.

3 The Problem Setup

In this section we describe the formal problem setup, and present a sequence of assumptions and lemmas which will be used in order to prove convergence of Algorithm 1 in section 4. These assumptions and lemmas mainly concern the properties of the controlled Markov chain, which represents the environment, and the properties of the actor’s parametrized policy function.

3.1 The Dynamics of the Environment and of the Actor

We consider an agent, composed of an actor and a critic, interacting with an environment. We model the environment as a Markov Decision Process (MDP) Puterman (1994) in discrete time with a finite state set $\mathcal{X}$ and an action set $\mathcal{U}$, which may be uncountable. We denote by $|\mathcal{X}|$ the size of the set $\mathcal{X}$. Each selected action $u\in U$ determines a stochastic matrix $P(u)=[P(y|x,u)]_{x,y\in\mathcal{X}}$ where $P(y|x,u)$ is the transition probability from a state $x\in\mathcal{X}$ to a state $y\in\mathcal{X}$ given the control $u$. For each state $x\in\mathcal{X}$ the agent receives a corresponding reward $r(x)$, which may be deterministic or random. In the present study we assume for simplicity that the reward is deterministic, a benign assumption which can be easily generalized.

Assumption 3.1

The rewards, $\{r(x)\}_{x\in\mathcal{X}}$, are uniformly bounded by a finite constant $B_{r}$.

The actor maintains a parametrized policy function. A parametrized policy function is a conditional probability function, denoted by $\mu(u|x,\theta)$, which maps an observation $x\in\mathcal{X}$ into a control $u\in\mathcal{U}$ given a parameter $\theta\in\mathbb{R}^{K}$. The agent’s goal is to adjust the parameter $\theta$ in order to attain maximum average reward over time. For each $\theta$, we have a Markov Chain (MC) induced by $P(y|x,u)$ and $\mu(u|x,\theta)$. The state transitions of the MC are obtained by first generating an action $u$ according to $\mu(u|x,\theta)$, and then generating the next state according to $\{P(y|x,u)\}_{x,y\in\mathcal{X}}$. Thus, the MC has a transition matrix $P(\theta)=[P(y|x,\theta)]_{x,y\in\mathcal{X}}$ which is given by

[TABLE]

We denote the space of these transition probabilities by $\mathcal{P}=\{P(\theta)|\theta\in\mathbb{R}^{K}\}$, and its closure by $\bar{\mathcal{P}}$. The following assumption is needed in the sequel in order to prove the main results (see Brémaud (1999) for definitions).

Assumption 3.2

Each MC, $P(\theta)\in\bar{\mathcal{P}}$, is aperiodic, recurrent, and irreducible.

As a result of Assumption 3.2, we have the following lemma regarding the stationary distribution and a common recurrent state.

Lemma 3.3

Under Assumption 3.2 we have:

Each MC, $P(\theta)\in\bar{\mathcal{P}}$, has a unique stationary distribution, denoted by $\pi(\theta)$, satisfying $\pi(\theta)^{\prime}P(\theta)=\pi(\theta)^{\prime}$. 2. 2.

There exists a state, denoted by $x^{*}$, which is recurrent for all $P(\theta)\in\bar{\mathcal{P}}$.

**Proof ** For the first part see Corollary 4.1 in (Gallager, 1995). The second part follows trivially from Assumption 3.2.  

The next technical assumption states that the first and second derivatives of the parametrized policy function are bounded, and is needed to prove Lemma 3.6 below.

Assumption 3.4

The conditional probability function $\mu(u|x,\theta)$ is twice differentiable. Moreover, there exist positive constants, $B_{\mu_{1}}$ and $B_{\mu_{2}}$, such that for all $x\in\mathcal{X}$, $u\in\mathcal{U}$, $\theta\in\mathbb{R}^{K}$ and $k_{1}\geq 1,\,k_{2}\leq K$ we have

[TABLE]

A notational comment concerning bounds Throughout the paper we denote upper bounds on different variables by the letter $B$, with a subscript corresponding to the variable itself. An additional numerical subscript, $1$ or $2$, denotes a bound on the first or second derivative of the variable. For example, $B_{f}$, $B_{f_{1}}$, and $B_{f_{2}}$ denote the bounds on the function $f$ and its first and second derivatives respectively.

3.2 Performance Measures

Next, we define a performance measure for an agent in an environment. The average reward per stage of an agent which traverses a MC starting from an initial state $x\in\mathcal{X}$ is defined by

[TABLE]

where $\textrm{E}[\cdot|\theta]$ denotes the expectation under the probability measure $P(\theta)$, and $x_{n}$ is the state at time $n$. The agent’s goal is to find $\theta\in\mathbb{R}^{K}$ which maximizes $J(x,\theta)$. The following lemma shows that under Assumption 3.2, the average reward per stage does not depend on the initial state (Bertsekas (2006), vol. II, section 4.1).

Lemma 3.5

Under Assumption 3.2 and based on Lemma 3.3, the average reward per stage, $J(x,\theta)$, is independent of the starting state, is denoted by $\eta(\theta)$, and satisfies $\eta(\theta)=\pi(\theta)^{\prime}r$.

Based on Lemma 3.5, the agent’s goal is to find a parameter vector $\theta$, which maximizes the average reward per stage $\eta(\theta)$. In the sequel we show how this maximization can be performed by optimizing $\eta(\theta)$, using $\nabla_{\theta}\eta(\theta)$. A consequence of Assumption 3.4 and the definition of $\eta(\theta)$ is the following lemma.

Lemma 3.6

**

For each $x,y\in\mathcal{X}$, $1\leq i,j\leq K$, and $\theta\in\mathbb{R}^{K}$, the functions $\partial P(y|x,\theta)/\partial\theta_{i}$ and $\partial^{2}P(y|x,\theta)/\partial\theta_{i}\partial\theta_{j}$ are uniformly bounded by $B_{P_{1}}$ and $B_{P_{2}}$ respectively.

1. (a)

For each $x\in\mathcal{X}$, $1\leq i,j\leq K$, and $\theta\in\mathbb{R}^{K}$, the functions $\partial\pi(x|\theta)/\partial\theta_{i}$ and $\partial^{2}\pi(x|\theta)/\partial\theta_{i}\partial\theta_{j}$ are uniformly bounded by , $B_{\pi_{1}}$and $B_{\pi_{2}}$ respectively. 2. (b)

For all $1\leq i,j\leq K$, and $\theta\in\mathbb{R}^{K}$, the functions $\eta(\theta)$, $\partial\eta(\theta)/\partial\theta_{i}$ and $\partial^{2}\pi(x|\theta)/\partial\theta_{i}\partial\theta_{j}$ are uniformly bounded by , $B_{\eta}$, $B_{\eta_{1}}$ and $B_{\eta_{2}}$ respectively. 3. (c)

For all $x\in\mathcal{X}$ and $\theta\in\mathbb{R}^{K}$, there exists a constant $b_{\pi}>0$ such that $\pi(x|\theta)\geq b_{\pi}$.

The proof is technical and is given in Appendix A.1. For later use, we define the random variable $T$, which denotes the first return time to the recurrent state $x^{*}$. Formally,

[TABLE]

It is easy to show that under Assumption 3.2, the average reward per stage can be expressed by

[TABLE]

Next, we define the differential value function of state $x\in\mathcal{X}$ which represents the average differential reward the agent receives upon starting from a state $x$ and reaching the recurrent state $x^{*}$ for the first time. Mathematically,

[TABLE]

Abusing notation slightly, we denote $h(\theta)\triangleq(h(x_{1},\theta),\ldots,h(x_{|\mathcal{X}|},\theta))\in\mathbb{R}^{|\mathcal{X}|}$. For each $\theta\in\mathbb{R}^{K}$ and $x\in\mathcal{X}$, $h(x,\theta)$, $r(x)$, and $\eta(\theta)$ satisfy Poisson’s equation (see Theorem 7.4.1 in (Bertsekas (2006))), i.e.,

[TABLE]

Based on the differential value we define the temporal difference (TD) between the states $x\in\mathcal{X}$ and $y\in\mathcal{X}$ (see Bertsekas and Tsitsiklis (1996), Sutton and Barto (1998)),

[TABLE]

According to common wisdom, the TD is interpreted as a prediction error. The next lemma states the boundedness of $h(x,\theta)$ and its derivatives. The proof is given in Appendix A.2.

Lemma 3.7

The differential value function, $h(x,\theta)$, is bounded and has bounded first and second derivative. Mathematically, for all $x\in\mathcal{X}$, $1\leq i,j\leq K$, and for all $\theta\in\mathbb{R}^{K}$ we have

[TABLE]

1. (a)

There exists a constant $B_{D}$ such that or all $\theta\in\mathbb{R}^{K}$ we have $\left|d(x,y,\theta)\right|\leq B_{D}$, where $B_{D}=2\left(B_{r}+B_{h}\right)$.

3.3 The Critic’s Dynamics

The critic maintains an estimate of the environmental state values. It does so by maintaining a parametrized function which approximates $h(x,\theta)$, and is denoted by $\tilde{h}(x,w)$. The function $\tilde{h}(x,w)$ is a function of the state $x\in\mathcal{X}$ and a parameter $w\in\mathbb{R}^{L}$. We note that $h(x,\theta)$ is a function of $\theta$, and is induced by the actor policy $\mu(u|x,\theta)$, while $\tilde{h}(x,w)$ is a function of $w$. Thus, the critic’s objective is to find the parameter $w$ which yields the best approximation of $h(\theta)=(h(x_{1},\theta),\ldots,h(x_{|\mathcal{X}|},\theta))$, in a sense to be defined later. We denote this optimal vector by $w^{*}(\theta)$. An illustration of the interplay between the actor, critic, and the environment is given in Figure 1.

4 A Single Time Scale Actor Critic Algorithm with Linear Function Approximation

In this section, we present a version of an AC algorithm, along with its convergence proof. The core of the algorithm is based on (7) below, where the actor’s estimate of $\nabla_{\theta}\eta(\theta)$ is based on the critic’s estimate of the TD signal $d(x,y,\theta)$. The algorithm is composed of three iterates, one for the actor and two for the critic. The actor maintains the iterate of the parameter vector $\theta$ corresponding to the policy $\mu(u|x,\theta)$, where its objective is to find the optimal value of $\theta$, denoted by $\theta^{*}$, which maximizes $\eta(\theta)$. The critic maintains the other two iterates. One iterate is used for estimating the average reward per stage, $\eta(\theta)$, where its estimate is denoted by $\tilde{\eta}$. The critic’s second iterate maintains a parameter vector, denoted by $w\in\mathbb{R}^{L}$, which is used for the differential value estimate using a function approximator, denoted by $\tilde{h}(w)$. For each $\theta\in\mathbb{R}^{K}$, there exists a $w^{*}(\theta)$ which, under the policy induced by $\theta$, is the optimal $w$ for estimating $\tilde{\eta}(w)$. The critic’s objective is to find the optimal $\tilde{\eta}$ and $w$.

4.1 Using the TD Signal to Estimate the Gradient of the Average Reward

We begin with a theorem which serves as the foundation for the policy gradient algorithm described in Section 4. The theorem relates the gradient of the average reward per stage, $\eta(\theta)$, to the TD signal. It was proved in (Bhatnagar et al. (2008a)), and is similar in its structure to other theorems which connect $\eta(\theta)$ to the $Q$-value (Konda and Tsitsiklis (2003)), and to the differential value function (Cao (2007); Marbach and Tsitsiklis (1998)).

We start with a definition of the likelihood ratio derivative

[TABLE]

where the gradient $\nabla_{\theta}$ is w.r.t.​ $\theta$, and $\psi(x,u,\theta)\in\mathbb{R}^{K}$. The following assumption states that $\psi(x,u,\theta)$ is bounded, and will be used to prove the convergence of algorithm 1.

Assumption 4.1

For all $x\in\mathcal{X}$, $u\in\mathcal{U}$, and $\theta\in\mathbb{R}^{K}$, there exists a positive constant, $B_{\psi}$, such that

[TABLE]

where $\|\cdot\|_{2}$ is the Euclidean $L_{2}$ norm.

Based on this, we present the following theorem which relates the gradient of $\eta(\theta)$ to the TD signal. For completeness, we supply a (straightforward) proof in Appendix B.

Theorem 4.2

For any arbitrary function $f(x)$, the gradient w.r.t. $\!\theta$ of the average reward per stage can be expressed by

[TABLE]

where $P(x,u,y,\theta)$ is the probability $\Pr(x_{n}=x,u_{n}=u,x_{n+1}=y)$ subject to the policy parameter $\theta$.

4.2 The updates performed by the critic and the actor

We note that the following derivation regarding the critic is similar in some respects to the derivation in section 6.3.3 of Bertsekas and Tsitsiklis (1996) and of Tsitsiklis and Roy (1997). We define the following quadratic target function used to evaluate the critic’s performance in assessing the differential value $h(\theta)$,

[TABLE]

The probabilities $\{\pi(x|\theta)\}_{x\in\mathcal{X}}$ are used in order to provide the proportional weight to the state estimates, according to the relative number of visits of the agent to the different states.

Limiting ourselves to the class of linear function approximations in the critic, we consider the following function for the differential value function

[TABLE]

where $\phi(x)\in\mathbb{R}^{L}$. We define $\Phi\in\mathbb{R}^{|\mathcal{X}|\times L}$ to be the matrix

[TABLE]

where $\phi(\cdot)$ is a column vector. Therefore, we can express (9) in vector form as

[TABLE]

where, abusing notation slightly, we set $\tilde{h}(w)=\left(\tilde{h}(x_{1},w),\ldots,\tilde{h}(x_{\left|\mathcal{X}\right|},w)\right)^{\prime}$.

We wish to express (8), and the approximation process, in an appropriate Hilbert space. Define the matrix $\Pi(\theta)$ to be a diagonal matrix $\Pi(\theta)\triangleq\textrm{diag}(\pi(\theta))$. Thus, (8) can be expressed as

[TABLE]

In the sequel, we will need the following technical assumption.

Assumption 4.3

The columns of the matrix $\Phi$ are independent, i.e., they form a basis of dimension $L$.

1. (a)

The norms of the column vectors of the matrix $\Phi$ are bounded above by $1$, i.e., $\|\phi_{k}\|_{2}\leq 1$ for $1\leq k\leq L$.

The parameter $w^{*}(\theta)$, which optimizes (11), can be directly computed, but involves inverting a matrix. Thus, in order to find the right estimate for $\tilde{h}(w)$, the following gradient descent (Bertsekas and Tsitsiklis (1996)) algorithm is suggested,

[TABLE]

where $\{\gamma_{n}\}_{n=1}^{\infty}$ is a positive series satisfying the following assumption, which will be used in proving the convergence of Algorithm 1.

Assumption 4.4

The positive series $\{\gamma_{n}\}_{n=1}^{\infty}$ satisfies

[TABLE]

Writing the term $\nabla_{w}I(w_{n})$ explicitly yields

[TABLE]

For each $\theta\in\mathbb{R}^{K}$, the value $w^{*}(\theta)$ is given by setting $\nabla_{w}I(w,\theta)=0$, i.e.,

[TABLE]

Note that Bertsekas and Tsitsiklis (1996) prove that the matrix $\left(\Phi^{\prime}\Pi(\theta)\Phi\right)^{-1}\Phi^{\prime}\Pi(\theta)$ is a projection operator into the space spanned by $\Phi w$, with respect to the norm $\left\|\cdot\right\|_{\Pi\left(\theta\right)}$ . Thus, the explicit gradient descent procedure (12) is

[TABLE]

Using the basis $\Phi$, in order to approximates $h\left(\theta\right)$, yields an approximation error defined by

[TABLE]

We can bound this error by

[TABLE]

The agent cannot access $h(x,\theta)$ directly. Instead, it can interact with the environment in order to estimate $h(x,\theta)$. We denote by $\hat{h}_{n}(x)$ the estimate of $h(x,\theta)$ at time step $n$, thus (16) becomes

[TABLE]

This procedure is termed stochastic gradient descent (Bertsekas and Tsitsiklis (1996)).

There exist several estimators for $\hat{h}_{n}$. One sound method, which performs well in practical problems (see Tesauro (1995)), is the TD($\lambda)$ method (see section 5.3.2 and 6.3.3 in Bertsekas and Tsitsiklis (1996), or Chapter 6 in Sutton and Barto (1998)), where the parameter $\lambda$ satisfies $0\leq\lambda\leq 1$. This method devises an estimator which is based on previous estimates of $h\left(w\right)$, i.e., $w_{n}$, and is based also on the environmental reward $r\left(x_{n}\right)$. This idea is a type of a bootstrapping algorithm, i.e., using existing estimates and new information in order to build more accurate estimates (see Sutton and Barto (1998), Section 6.1).

The TD$\left(\lambda\right)$ estimator for $\hat{h}_{n+1}$ is

[TABLE]

where the* $k$-steps predictor* is defined by

[TABLE]

The idea of bootstrapping is apparent in (19): the predictor for the differential value of the state $x_{n}$ at the $\left(n+1\right)$-Th time step, is based partially on the previous estimates through $\hat{h}_{n}\left(x_{n+k+1}\right)$, and partially on new information, i.e., the reward $r\left(x_{n+m}\right)$. In addition, the parameter $\lambda$ gives an exponential weighting for the different $k$-step predictors. Thus, choosing the right $\lambda$ can yield better estimators.

For the discounted setting, it was proved by Bertsekas and Tsitsiklis (1996) (p. 295) that an algorithm which implements the TD$\left(\lambda\right)$ estimator (19) online and converges to the right value is the following one

[TABLE]

where $d_{n}$ is the temporal difference between the $n$-th and the $\left(n+1\right)$-th cycle, and $e_{n}$ is the so-called eligibility trace (see Sections 5.3.3 and 6.3.3 inBertsekas and Tsitsiklis (1996) or Chapter 7 in Sutton and Barto (1998)), and the parameter $\alpha$ is the discount factor. The eligibility trace is an auxiliary variable, which is used in order to implement the idea of (19) as an online algorithm. As the name implies, the eligibility variable measures how eligible is the TD variable, $d_{n}$, in (20).

In our setting, the non-discounted case, the analogous equations for the critic, are

[TABLE]

The actor’s iterate is motivated by Theorem 4.2. Similarly to the critic, the actor executes a stochastic gradient ascent step in order to fior with a parametrized policy $\mu(u|x,\theta)$ satisfying Assumptions 3.4 and 4.1.

• •

A critic with

nd a local maximum of the average reward per stage $\eta(\theta)$. Therefore,

[TABLE]

A summary of the algorithm is presented in Algorithm 1.

4.3 Convergence Proof for the AC Algorithm

In the remainder of this section, we state the main theorems related to the convergence of Algorithm 1. We present a sketch of the proof in this section, where the technical details are relegated to Appendices C and D. The proof is divided into two stages. In the first stage we relate the stochastic approximation to a set of ordinary differential equations (ODE). In the second stage, we find conditions under which the ODE system converges to a neighborhood of the optimal $\eta(\theta)$.

The ODE approach is a widely used method in the theory of stochastic approximation for investigating the asymptotic behavior of stochastic iterates, such as (23)-(25). The key idea of the technique is that the iterate can be decomposed into a mean function and a noise term, such as a martingale difference noise. As the iterates advance, the effect of the noise weakens due to repeated averaging. Moreover, since the step size of the iterate decreases (e.g., $\gamma_{n}$ in (23)-(25)), one can show that asymptotically an interpolation of the iterates converges to a continuous solution of the ODE. Thus, the first part of the convergence proof is to find the ODE system which describes the asymptotic behavior of Algorithm 1. This ODE will be presented in Theorem 4.6. In the second part we use ideas from the theory of Lyapunov functions in order to characterize the relation between the constants, $|\mathcal{X}|$, $\Gamma_{\eta}$, $\Gamma_{w}$, etc., which ensure convergence to some neighborhood of the maximum point satisfying $\|\nabla_{\theta}\eta(\theta)\|_{2}=0$. Theorem 4.7 states conditions on this convergence.

4.3.1 Relate the Algorithm to an ODE

In order to prove the convergence of this algorithm to the related ODE, we need to introduce the following assumption, which adds constraints to the iteration for $w$, and will be used in the sequel to prove Theorem 4.6. This assumption may seem restrictive at first but in practice it is not. The reason is that we usually assume the bounds of the constraints to be large enough so the iterates practically do not reach those bounds. For example, under Assumption 3.2 and additional mild assumptions, it is easy to show that $h(\theta)$ is uniformly bounded for all $\theta\in\mathbb{R}^{K}$. As a result, there exist a constant bounding $w^{*}(\theta)$ for all $\theta\in\mathbb{R}^{K}$. Choosing constraints larger than this constant will not influence the algorithm performance.

Definition 4.5

Let us denote by $\{w_{i}\}_{i=1}^{L}$ the components of $w$, and choose a positive constant $B_{w}$. We define the set $H\subset\mathbb{R}^{K}\times\mathbb{R}^{L}$ to be

[TABLE]

and let $\Psi_{w}$ be an operator which projects $w$ onto $H$, i.e., for each $Cramer^{\prime}s1\leq j\leq L$, $\Psi_{w}w_{j}=\max(\min(w_{j},B_{w}),-B_{w})$.

The following theorem identifies the ODE system which corresponds to Algorithm 1. The detailed proof is given in Appendix C.

Theorem 4.6

Define the following functions:

[TABLE]

Then,

Algorithm 1 converges to the invariant set of the following set of ODEs

[TABLE]

with probability 1.

1. (a)

The functions in (26) are continuous with respect to $\theta$.

4.3.2 Investigating the ODE Asymptotic Behavior

Next, we quantify the asymptotic behavior of the system of ODEs in terms of the various algorithmic parameters. The proof of the theorem appears in Appendix D.

Theorem 4.7

Consider the constants $\Gamma_{\eta}$ and $\Gamma_{w}$ as defined in Algorithm 1, and the function approximation bound $\epsilon_{\textrm{app}}$ as defined in (17). Setting

[TABLE]

where $B_{\Delta td1}$, $B_{\Delta td2}$, $B_{\Delta td3}$ are a finite constants depending on the MDP and agent parameters. Then, the ODE system (27) satisfies

[TABLE]

Theorem 4.7 has a simple interpretation. Consider the trajectory $\eta(\theta_{t})$ for large times, corresponding to the asymptotic behavior of $\eta_{n}$. The result implies that the trajectory visits a neighborhood of a local maximum infinitely often. Although it may leave the local vicinity of the maximum, it is guaranteed to return to it infinitely often. This occurs, since once it leaves the vicinity, the gradient of $\eta$ points in a direction which has a positive projection on the gradient direction, thereby pushing the trajectory back to the vicinity of the maximum. It should be noted that in simulation (reported below) the trajectory usually remains within the vicinity of the local maximum, rarely leaving it. We also observe that by choosing appropriate values for $\Gamma_{\eta}$ and $\Gamma_{w}$ we can control the size of the ball to which the algorithm converges.

The key idea required to prove the Theorem is the following argument. If the trajectory does not satisfy $\|\nabla\eta(\theta)\|_{2}\leq B_{\nabla\eta}$, we have $\dot{\eta}(\theta)>\epsilon$ for some positive $\epsilon$. As a result, we have a monotone function which increases to infinity, thereby contradicting the boundedness of $\eta(\theta)$. Thus, $\eta(\theta)$ must visit the set which satisfies $\|\nabla\eta(\theta)\|_{2}\leq B_{\nabla\eta}$ infinitely often.

5 A Comparison to other convergence results

In this section, we point out the main differences between Algorithm 1, the first algorithm proposed by Bhatnagar et al. (2008b) and the algorithms proposed by Konda and Tsitsiklis (2003). The main dimensions along which we compare the algorithms are the time scale, the type of the TD signal, and whether the algorithm is on line or off line.

The Time Scale and Type of Convergence

As was mentioned previously, the algorithms of Bhatnagar et al. (2008b) and Konda and Tsitsiklis (2003) need to operate in two time scales. More precisely, this refers to the following situation. Denote the time step of the critic’s iteration by $\gamma_{n}^{c}$ and the time step of the actor’s iteration by $\gamma_{n}^{a}$, we have $\gamma_{n}^{c}=o(\gamma_{n}^{a})$, i.e.,

[TABLE]

The use of two time scales stems from the need of the critic to give an accurate estimate of the state values (as in the work of Bhatnagar et al. (2008b)) or the state-action values (as in the work of Konda and Tsitsiklis (2003)) before the actor uses them.

In the algorithm proposed here, a single time scale is used for the three iterates of Algorithm 1. We have $\gamma_{n}^{a}=\gamma_{n}$ for the actor iterate, $\gamma_{n}^{c,\eta}=\Gamma_{\eta}\gamma_{n}$ for the critic’s $\eta_{n}$ iterate, and $\gamma_{n}^{c,w}=\Gamma_{w}\gamma_{n}$ for the critic’s $w$ iterate. Thus,

[TABLE]

Due to the single time scale, Algorithm 1 has the potential to converge faster than algorithms based on two time scales, since both the actor and the critic may operate on the fast time scale. The drawback of Algorithm 1 is the fact that convergence to the optimal value cannot be guaranteed, as was proved by Bhatnagar et al. (2008b) and by Konda and Tsitsiklis (2003). Instead, convergence to a neighborhood in $\mathbb{R}^{K}$ around the optimal value is guaranteed. In order to make the neighborhood smaller, we need to choose $\Gamma_{\eta}$ and $\Gamma_{w}$ appropriately, as is stated in Theorem 4.7.

The TD Signal, the Information Passed Between the Actor and the Critic,

and the Critic’s Basis

The algorithm presented in Bhatnagar et al. (2008b) is essentially a TD(0) algorithm, while the algorithm in Konda and Tsitsiklis (2003) is TD(1), Our algorithm is a TD$\left(\lambda\right)$ for $0\leq\lambda<1$. A major difference between the approaches in Bhatnagar et al. (2008b) and the present work, as compared to Konda and Tsitsiklis (2003), is the information passed from the critic to the actor. In the former cases, the information passed is the TD signal, while in the latter case the Q-value is passed. Additionally, in Bhatnagar et al. (2008b) and in Algorithm 1 the critic’s basis functions do not change through the simulation, while in Konda and Tsitsiklis (2003) the critic’s basis functions are changed in each iteration according to the actor’s parameter $\theta$. Finally, we comment that Bhatnagar et al. (2008b) introduced an additional algorithm, based on the so-called natural gradient, which led to improved convergence speed. In this work we limit ourselves to algorithms based on the regular gradient, and defer the incorporation of the natural gradient to future work. As stated in Section 1, our motivation in this work was the derivation of a single time scale online AC algorithm with guaranteed convergence, which may be applicable in a biological context. The more complex natural gradient approach seems more restrictive in this setting.

6 Simulations

We report empirical results applying Algorithm 1 to a set of abstract randomly constructed MDPs which are termed Average Reward Non-stationary Environment Test-bench or in short garnet (Archibald et al. (1995)). garnet problems comprise a class of randomly constructed finite MDPs serving as a test-bench for control and RL algorithms optimizing the average reward per stage. A garnet problem is characterized in our case by four parameters and is denoted by garnet$(X,U,B,\sigma)$. The parameter $X$ is the number of states in the MDP, $U$ is the number of actions, $B$ is the branching factor of the MDP, i.e., the number of non-zero entries in each line of the MDP’s transition matrices, and $\sigma$ is the variance of each transition reward.

We describe how a garnet problem is generated. When constructing such a problem, we generate for each state a reward, distributed normally with zero mean and unit variance. For each state-action the reward is distributed normally with the state’s reward as mean and variance $\sigma^{2}$. The transition matrix for each action is composed of $B$ non-zero terms in each line which sum to one.

We note that a comparison was carried out by Bhatnagar et al. (2008b) between their algorithm and the algorithm of Konda and Tsitsiklis (2003). We therefore compare our results directly to the more closely related former approach (see also Section 5).

We consider the same garnet problems as those simulated by Bhatnagar et al. (2008b). For completeness, we provide here the details of the simulation. For the critic’s feature vector, we use a linear function approximation $\tilde{h}(x,w)=\phi(x)^{\prime}w$, where $\phi(x)\in\{0,1\}^{L}$, and define $l$ to be the number nonzero values in $\phi(x)$. The nonzero values are chosen uniformly at random, where any two states have different feature vectors. The actor’s feature vectors are of size $L\times|\mathcal{U}|$, and are constructed as

[TABLE]

Bhatnagar et al. (2008b) reported simulation results for two garnet problems: garnet$(30,4,2,0.1)$ and garnet$(100,10,3,0.1)$. For the garnet$(30,4,2,0.1)$ problem, Bhatnagar et al. (2008b) used critic steps $\gamma_{n}^{c,w}$ and $\gamma_{n}^{c,\eta}$, and actor steps $\gamma_{n}^{a}$, where

[TABLE]

and for garnet$(100,10,3,0.1)$ the steps were

[TABLE]

In our simulations we used a single time scale, $\gamma_{n},$ which was equal to $\gamma_{n}^{c,w}$ as used by Bhatnagar et al. (2008b). The basis parameters for garnet$(30,4,2,0.1)$ were $L=8$ and $l=3$, where for garnet$(100,10,3,0.1)$ they were $L=20$ and $l=5$.

In Figures 2 we show results of applying Algorithm 1 (solid line) and algorithm 1 from Bhatnagar et al. (2008b) (dashed line) on garnet$(30,4,2,0.1)$ and garnet$(100,10,3,0.1)$ problems. Each graph in Figure 2, represents an average of $100$ independent simulations. Note that an agent with a uniform action selection policy will attain an average reward per stage of zero in these problems. Figure 3 presents similar results for garnet$(30,15,15,0.1)$. We see from these results that in all simulations, during the initial phase, Algorithm 1 converges faster than algorithm 1 from Bhatnagar et al. (2008b). The long term behavior is problem-dependent, as can be seen by comparing figures 2 and 3; specifically, in Figure 2 the present algorithm converges to a higher value than Bhatnagar et al. (2008b), while the situation is reversed in Figure 3. We refer the reader toMokkadem and Pelletier (2006) for careful discussion of convergence rates for two time scales algorithms; a corresponding analysis of convergence rates for single time scale algorithms is currently an open problem.

The results displayed here suggest a possible avenue for combining both algorithms. More concretely, using the present approach may lead to faster initial convergence due to the single time scale setting, which allows both the actor and the critic to evolve rapidly, while switching smoothly to a two time scales approach as in (Bhatnagar et al. (2008b)) will lead to asymptotic convergence to a point rather than to a region. This type of approach is reminiscent of the quasi-Newton algorithms in optimization, and is left for future work. As discussed in Section 5, we do not consider the natural gradient based algorithms from Bhatnagar et al. (2008b) in this comparative study.

7 Discussion and Future Work

We have introduced an algorithm where the information passed from the critic to the actor is the temporal difference signal, while the critic applies a $\textrm{TD}(\lambda)$ procedure. A policy gradient approach was used in order to update the actor’s parameters, based on a critic using linear function approximation. The main contribution of this work is a convergence proof in a situation where both the actor and the critic operate on the same time scale. The drawback of the extra flexibility in time scales is that convergence is only guaranteed to a neighborhood of a local maximum value of the average reward per stage. However, this neighborhood depends on parameters which may be controlled to improve convergence.

This work sets the stage for much future work. First, as observed above, the size of the convergence neighborhood is inversely proportional to the step sizes $\Gamma_{w}$ and $\Gamma_{\eta}$. In other words, in order to reduce this neighborhood we need to select larger values of $\Gamma_{w}$ and $\Gamma_{\eta}$. This on the other hand increases the variance of the algorithm. Therefore, further investigation of methods which reduce this variance are needed. However, the bounds used throughout are clearly rather loose, and cannot be effectively used in practical applications. Obviously, improving the bounds, and conducting careful numerical simulations in order to obtain a better practical understanding of the influence of the different algorithmic parameters, is called for. In addition, there is clearly room for combining the advantages of our approach with those of AC algorithms for which convergence to a single point is guaranteed, as discussed in Section 6,

From a biological point of view, our initial motivation to investigate TD based AC algorithms stemmed from questions related to the implementation of RL in the mammalian brain. Such a view is based on an interpretation of the transient activity of the neuromodulator dopamine as a TD signal (e.g., Schultz (2002)). Recent evidence suggested that the dorsal and ventral striatum may implement the actor and the critic, respectively (e.g., Daw et al. (2006)). We believe that theoretical models such as (Bhatnagar et al. (2008b)) and Algorithm 1 may provide, even if partially, a firm foundation to theories at the neural level. Some initial attempts in a neural setting (using direct policy gradient rather than AC based approaches) have been made by Baras and Meir (2007) and Florian (2007). Such an approach may lead to functional insights as to how an AC paradigm may be implemented at the cellular level of the basal ganglia and cortex. An initial demonstration was given by DiCastro et al. (2008).

From a theoretical perspective many issues remain open. First, strengthening Theorem (4.7) by replacing $\liminf$ by $\lim$ would clearly be useful. Second, extending the recent convergence rate results in Mokkadem and Pelletier (2006) to the single time scale case is an important challenging problem. Third, systematically combining the advantages of single time scale convergence (fast initial dynamics) and two time scales approaches (convergence to a point) would clearly be beneficial.

Acknowledgment The authors are grateful to Mohammad Ghavamzadeh for sending them a copy of Bhatnagar et al. (2008b) prior to publication, and to the anonymous reviewers for their helpful comments. The work of R. Meir was partially supported by an ISF Converging Technologies grant, and by ISF grant 665/08.

**APPENDIX **

A Proofs of Results from Section 3

A.1 Proof of Lemma 3.6

Looking at (1) we see that $P(y|x,\theta)$ is a compound function of an integral and a twice differentiable function, $\mu(y|x,\theta)$, with bounded first and second derivatives according to Assumption 3.4. Therefore, $P(y|x,\theta)$ is a twice differentiable function with bounded first and second derivatives for all $\theta\in\mathbb{R}^{K}$. 2. 2.

According to Lemma 3.3, for each $\theta\in\mathbb{R}^{K}$ we have a unique solution to the following non-homogeneous linear equation system in $\{\pi(i|\theta)\}_{i=1}^{|\mathcal{X}|}$,

[TABLE]

or in matrix form $M(\theta)\pi(\theta)=b$. By Assumption 3.2, the equation system (29) is invertible, therefore, $\det[M(\theta)]>0$. This holds for all $P(\theta)\in\bar{P}$, thus, there exists a positive constant, $b_{M}$, which uniformly lower bounds $\det[M(\theta)]$ for all $\theta\in\mathbb{R}^{K}$.Thus, using Cramer’s rule we have

[TABLE]

where $Q(i,\theta)$ is a finite polynomial of $\{P(j|i,\theta)\}_{i,j\in\mathcal{X}}$ of at most degree $\left|\mathcal{X}\right|$ and with at most $\left|\mathcal{X}\right|!$ terms. Writing $\partial\pi(x|\theta)/\partial\theta_{i}$ explicitly gives

[TABLE]

which gives the desired bound. Following similar steps we can show the boundedness of the second derivatives. 3. 3.

The average reward per stage, $\eta(\theta)$ is a linear combination of $\{\pi(i|\theta)\}_{i=1}^{|\mathcal{X}|}$, with bounded coefficients by assumption 3.1. Therefore, using section 2, $\eta(\theta)$ is twice differentiable with bounded first and second derivatives for all $\theta\in\mathbb{R}^{K}$. 4. 4.

Since $\pi(x|\theta)$ is the stationary distribution of a recurrent MC, according to Assumption 3.2 there is a positive probability to be in each state $x\in\mathcal{X}$. This applies to the closure of $\mathcal{P}$. Thus, there exist a positive constant $b_{\pi}$ such that $\pi(x|\theta)\geq b_{\pi}$.

A.2 Proof of Lemma 3.7

We recall the Poisson equation (5). We have the following system of linear equations in $\{h(x|\theta)\}_{x\in\mathcal{X}}$, namely,

[TABLE]

or in matrix form $N(\theta)h(\theta)=c$. Adding the equation $h(x^{*}|\theta)=0$ yields a unique solution for the system (Bertsekas (2006), Vol. 1, Prop. 7.4.1). Thus, using Cramer’s rule we have $h(x|\theta)=R(x,\theta)/\det[N(\theta)]$, where $R(x,\theta)$ and $\det[N(\theta)]$ are polynomial function of entries in $N(\theta)$, which are bounded and have bounded first and second derivatives according to Lemma 3.6. Continuing in the same steps of Lemma 3.6 proof, we conclude that $h(x|\theta)$ and its two first derivatives for all $x\in\mathcal{X}$ and for all $\theta\in\mathbb{R}^{K}$. 2. 2.

Trivially, by (6) and the previous section the result follows.

B Proof of Theorem 4.2

We begin with a Lemma which was proved in (Marbach and Tsitsiklis (1998)). It relates the gradient of the average reward per stage to the differential value function.

Lemma B.1

The gradient of the average reward per stage can be expressed by

[TABLE]

For completeness, we present a proof,which will be used in the sequel.

**Proof ** We begin with Poisson’s equation (5) in vector form

[TABLE]

where $e$ is a column vector of $1$’s. Taking the derivative with respect to $\theta$ and rearranging yields

[TABLE]

Multiplying the left hand side of the last equation by the stationary distribution $\pi(\theta)^{\prime}$ yields

[TABLE]

Expressing the result explicitly we obtain

[TABLE]

Based on this, we can now prove Theorem 4.2. We start with the result in (32).

[TABLE]

In order to complete the proof, we show that the second term equals [math]. We define $F(x,\theta)\triangleq-h(x|\theta)+\bar{r}(x)-\eta(\theta)+f(x)$ and obtain

[TABLE]

C Proof of Theorem 4.6

As mentioned earlier, we use Theorem 6.1.1 of Kushner and Yin (1997). We start by describing the setup of the theorem and the main result. Then, we show that the required assumptions hold in our case.

C.1 Setup, Assumptions and Theorem 6.1.1 of Kushner and Yin (1997).

In this section we describe briefly but accurately the conditions for Theorem 6.1.1 of Kushner and Yin (1997) and state the main result. We consider the following stochastic iteration

[TABLE]

where $Y_{n}$ is a vector of “observations” at time $n$, and $\Pi_{H}$ is a constraint operator as defined in Definition 4.5. Recall that $\left\{x_{n}\right\}$ is a Markov chain. Based on this, define $\mathcal{F}_{n}$ to be the $\sigma$-algebra

[TABLE]

and

[TABLE]

The difference between the $\sigma$-algebras is the sequence $\left\{x_{n}\right\}$. Define the conditioned average iterate

[TABLE]

and the corresponding martingale difference noise

[TABLE]

Thus, we can write the iteration as

[TABLE]

where $Z_{n}$ is a reflection term which forces the iterate to the nearest point in the set $H$ whenever the iterates leaves it (see Kushner and Yin (1997) for details). Next, set

[TABLE]

Later, we will see that the sum of the sequence $\left\{\delta M_{n}\right\}$ converges to [math], and the r.h.s of the iteration behaves approximately as a the function $\bar{g}\left(y\right)$, which yields the corresponding ODE, i.e.,

[TABLE]

The following ODE method will show that the asymptotic behavior of the iteration is equal to the asymptotic behavior of the corresponding ODE.

Define the auxiliary variable

[TABLE]

and the monotone piecewise constant auxiliary function

[TABLE]

The following assumption, taken from Section 6.1 of Kushner and Yin (1997), is required to establish the basic Theorem. An interpretation of the assumption follows its statement.

Assumption C.1

Assume that

The coefficients $\left\{\gamma_{n}\right\}$ satisfy $\sum_{n=1}^{\infty}\gamma_{n}=\infty$ and $\lim_{n\rightarrow\infty}\gamma_{n}=0$.

1. (a)

$\sup_{n}\textrm{E}\left[\left\|Y_{n}\right\|\right]<\infty.$ 2. (b)

$g_{n}\left(y_{n},x\right)$ is continuous in $y_{n}$ for each $x$ and $n$. 3. (c)

For each $\mu>0$ and for some $T>0$ there is a continuous function $\bar{g}\left(\cdot\right)$ such that for each $y$

[TABLE] 4. (d)

For each $\mu>0$ and for some $T>0$ we have

[TABLE] 5. (e)

There are measurable and non-negative functions $\rho_{3}\left(y\right)$ and $\rho_{n4}\left(x\right)$such that

[TABLE]

where $\rho_{3}\left(y\right)$ is bounded on each bounded $y$-set , and for each $\mu>0$ we have

[TABLE] 6. (f)

There are measurable and non-negative functions $\rho_{1}\left(y\right)$ and $\rho_{n2}\left(x\right)$such that $\rho_{1}\left(y\right)$ is bounded on each bounded $y$-set and

[TABLE]

where

[TABLE]

and

[TABLE]

The conditions of Assumption C.1 are quite general but can be interpreted as follows. Assumptions C.1.1-3 are straightforward. Assumption C.1.4 is reminiscent of ergodicity, which is used to replace the state-dependent function $g_{n}\left(\cdot,\cdot\right)$ with the state-independent of state function $\bar{g}\left(\cdot\right)$, whereas Assumption C.1.5 states that the martingale difference noise converges to [math] in probability. Assumptions C.1.6 and C.1.7 ensure that the function $g_{n}\left(\cdot,\cdot\right)$ is not unbounded and satisfies a Lipschitz condition.

The following Theorem, adapted from Kushner and Yin (1997), provides the main convergence result required. The remainder of this appendix shows that the required conditions in Assumption C.1 hold.

Theorem C.2

(Adapted from Theorem 6.1.1 in Kushner and Yin (1997)) Assume that algorithm 1, and Assumption C.1 hold. Then $y_{n}$ converges to some invariant set of the projected ODE

[TABLE]

Thus, the remainder of this section is devoted to showing that Assumptions C.1.1-C.1.7 are satisfied.

For future purposes, we express Algorithm 1 using the augmented parameter vector $y_{n}$

[TABLE]

The components of $Y_{n}$ are determined according to (27). The corresponding sub-vectors of $\bar{g}(y_{n})$ will be denoted by

[TABLE]

and similarly

[TABLE]

We begin by examining the components of $g_{n}\left(y_{n},x_{n}\right)$ and $\bar{g}\left(y_{n}\right)$. The iterate $g_{n}\left(\tilde{\eta}_{n},x_{n}\right)$ is

[TABLE]

and since there is no dependence on $x_{n}$ we have also

[TABLE]

The iterate $g_{n}\left(w_{n},x_{n}\right)$ is

[TABLE]

and the iterate $\bar{g}\left(w_{n}\right)$ is

[TABLE]

which, following, Bertsekas and Tsitsiklis (1996) section 6.3, can be written in matrix form

[TABLE]

With some further algebra we can express this using (26),

[TABLE]

Finally, the iterate $g_{n}\left(\theta_{n},x_{n}\right)$ is

[TABLE]

and

[TABLE]

Next, we show that the required assumptions hold.

C.2 Satisfying Assumption C.1.2

We need to show that $\sup_{n}\textrm{E}\left[\left\|Y_{n}\right\|_{2}\right]<\infty.$ Since later we need to show that $\sup_{n}\textrm{E}\left[\left\|Y_{n}\right\|_{2}^{2}\right]<\infty$, and the proof of the second moment is similar to the proof of the first moment, we consider both moments here.

Lemma C.3

The sequence $\tilde{\eta}_{n}$ is bounded w.p. $\!1$, $\sup_{n}\mathrm{\textrm{E}}\left[\left\|Y_{n}\left(\tilde{\eta}_{n}\right)\right\|_{2}\right]<\infty$, and $\sup_{n}\mathrm{\textrm{E}}\left[\left\|Y_{n}\left(\tilde{\eta}_{n}\right)\right\|_{2}^{2}\right]<\infty$

**Proof ** We can choose $M$ such that $\gamma_{n}\Gamma_{\eta}<1$ for all $n>M$. Using Assumption 3.4 for the boundedness of the rewards, we have

[TABLE]

which means that each iterate is bounded above by the previous iterate or by a constant. We denote this bound by $B_{\tilde{\eta}}$. Using similar arguments we can prove that $\tilde{\eta}_{n}$ is bounded below, and the first part of the lemma is proved. Since $\tilde{\eta}_{n+1}$ is bounded the second part follows trivially.

Lemma C.4

We have $\sup_{n}\textrm{E}\left[\left\|Y_{n}\left(w_{n}\right)\right\|_{2}^{2}\right]<\infty$ and $\sup_{n}\textrm{E}\left[\left\|Y_{n}\left(w_{n}\right)\right\|_{2}\right]<\infty$

**Proof ** For the first part we have

[TABLE]

where we used the triangle inequality in $(a)$ and the inequality $(a+b)^{2}\leq 2a^{2}+2b^{2}$ in $(b)$. The bound $\sup_{n}\textrm{E}\left[\left\|Y_{n}\left(w_{n}\right)\right\|_{2}\right]<\infty$ follows directly from the Cauchy-Schwartz inequality.

Lemma C.5

We have $\sup_{n}\textrm{E}\left[\left\|Y_{n}\left(\theta_{n}\right)\right\|_{2}^{2}\right]<\infty$ and $\sup_{n}\textrm{E}\left[\left\|Y_{n}\left(\theta_{n}\right)\right\|_{2}\right]<\infty$.The proof proceeds as in Lemma C.4.

Based on Lemmas C.3, C.4, and C.5 we can assert Assumption C.1.2

C.3 Satisfying Assumption C.1.3

Assumption C.1.3 requires the continuity of $g_{n}\left(y_{n},x_{n}\right)$ for each $n$ and $x_{n}$. Again, we show that this assumption holds for the three parts of the vector $y_{n}$.

Lemma C.6

The function $g_{n}\left(\tilde{\eta}_{n},x_{n}\right)$ is a continuous function of $\tilde{\eta}_{n}$ for each $n$ and $x_{n}$.

**Proof ** Since $g_{n}\left(\tilde{\eta}_{n},x_{n}\right)=\Gamma_{\eta}\left(r\left(x_{n}\right)-\tilde{\eta}_{n}\right)$ the claim follows.

Lemma C.7

The function $g_{n}\left(w_{n},x_{n}\right)$ is a continuous function of $\tilde{\eta}_{n}$, $w_{n}$, and $\theta_{n}$ for each $n$ and $x_{n}$.

**Proof ** The function is

[TABLE]

The probability transition $\sum_{y\in\mathcal{X}}P\left(y|x_{n},\theta_{n}\right)$ can be written as $\sum_{y\in\mathcal{X},u\in\mathcal{U}}P\left(y|x_{n},u_{n}\right)\mu\left(u_{n}|x_{n},\theta_{n}\right)$. The function $\mu\left(u_{n}|x_{n},\theta_{n}\right)$ is continuous in $\theta_{n}$ by Assumption 3.6, and thus $g_{n}\left(w_{n},x_{n}\right)$ is continuous in $\tilde{\eta}_{n}$ and $\theta_{n}$ and the lemma follows.

Lemma C.8

The function $g_{n}\left(\theta_{n},x_{n}\right)$ is a continuous function of $\tilde{\eta}_{n}$, $w_{n}$, and $\theta_{n}$ for each $n$ and $x_{n}$.

**Proof ** By definition, the function $g_{n}\left(\theta_{n},x_{n}\right)$ is

[TABLE]

Using similar arguments to Lemma C.7 the claim holds.

C.4 Satisfying Assumption C.1.4

In this section we prove the following convergence result: for each $\mu>0$ and for some $T>0$ there is a continuous function $\bar{g}\left(\cdot\right)$ such that for each $y$

[TABLE]

We start by showing that there exist independent cycles of the algorithm since the underlying Markov chain is recurrent and aperiodic. Then, we show that the cycles behave as a martingale, thus Doob’s inequality can be used. Finally we show that the sum in (39) converges to [math] w.p. 1. We start investigating the regenerative nature of the process.

Based on Lemma 3.2, there exists a recurrent state common to all $MC(\theta)$, denoted by $x^{*}$. We define the series of hitting times of the recurrent state $x^{*}$ by $t_{0}=0,t_{1},t_{2},...$, where $t_{m}$ it the $m$-th time the agent hits the state $x^{*}$. Mathematically, we can define this series recursively by

[TABLE]

and $T_{m}\triangleq t_{m+1}-t_{m}$. Define the $m$-th cycle of the algorithm to be the set of times

[TABLE]

and the corresponding trajectories

[TABLE]

Define a function, $\varrho\left(k\right)$, which returns the cycle to which the time $k$ belongs to, i.e.,

[TABLE]

We notice that based on Lemma 3.3, and using the Regenerative Cycle Theorem (see Brémaud (1999), pp. 87), the cycles $\mathcal{C}_{m}$ are independent of each other.

Next, we examine (39), and start by defining the following events:

[TABLE]

It is easy to show that for each $n$ we have $b_{n}^{\left(1\right)}\subset b_{n}^{\left(2\right)},$ thus,

[TABLE]

It is easy to verify that the series $\left\{b_{n}^{\left(2\right)}\right\}$ is a subsequence of $\left\{b_{n}^{\left(3\right)}\right\}$. Thus, if we prove that $\lim_{n\rightarrow\infty}\Pr\left(b_{n}^{\left(3\right)}\right)=0$, then $\lim_{n\rightarrow\infty}\Pr\left(b_{n}\right)=0$, and using (42), Assumption C.1.4 holds.

Next, we examine the sum defining the event $b_{n}^{\left(3\right)}$, by splitting it a sum over cycles and a sum within each cycle. We can write it as following

[TABLE]

Denote $c_{m}\triangleq\sum_{j\in\mathcal{T}_{m}}\gamma_{i}\left(g_{n}\left(y,x_{i}\right)-\bar{g}\left(y\right)\right)$. Therefore, by the Regenerative Cycle Theorem (Brémaud (1999), pp. 87), $c_{m}$ are independent random variables. Also,

[TABLE]

We argue that $c_{m}$ is square integrable. To prove this we need to show that the second moments of $T_{m}$ and $\left(g_{n}\left(y,x_{i}\right)-\bar{g}\left(y\right)\right)$ are finite.

Lemma C.9

The first two moments of the random times $\left\{T_{m}\right\}$ are bounded above by a constant $B_{T}$, for all $\theta\in\mathbb{R}^{K}$ and for all $m$, $1\leq m<\infty.$

1. (a)

$\textrm{E}\left[\left(g_{n}\left(y,x_{i}\right)-\bar{g}\left(y\right)\right)^{2}\right]\leq B_{g}$ 2. (b)

Define $\bar{\gamma}{}_{m}\triangleq\sup_{i\in\mathcal{T}_{m}}\gamma_{i}$, then $\sum_{m=0}^{\infty}\bar{\gamma}{}_{m}^{2}<\infty$. 3. (c)

$\textrm{E}\left[c_{m}^{2}\right]\leq\left(B_{T}B_{g}\right)^{2}$.

**Proof **

According to Assumption 3.2 and Lemma 3.3, each Markov chain in $\bar{\mathcal{P}}$ is recurrent. Thus, for each $\theta\in\mathbb{R}^{K}$ there exists a constant $\tilde{B}_{T}(\theta)$, $0<\tilde{B}_{T}(\theta)<1$, where for $k\leq\left|\mathcal{X}\right|$ we have

[TABLE]

where $\lfloor a\rfloor$ is the largest integer which is not greater than $a$. Otherwise, if for $k>\left|\mathcal{X}\right|$ we have $\tilde{B}_{T}(\theta_{m})=1$ then the chain transitions equal $1$ which contradicts the aperiodicity of the chains. Therefore,

[TABLE]

and

[TABLE]

Since the set $\bar{\mathcal{P}}$ is closed, by Assumption 3.2 the above holds for the closure of $\bar{\mathcal{P}}$ as well. Thus, there exists a constant $B_{T}$ satisfying $B_{T}=\max\{\sup_{\theta}B_{T_{1}}(\theta),\sup_{\theta}B_{T_{2}}(\theta)\}<\infty$.

1. (a)

The proof proceeds along the same lines as the proofs of lemmas C.3, C.4, and C.5. 2. (b)

The result follows trivially since the sequence $\left\{\bar{\gamma}{}_{m}\right\}$ is subsequence of the summable sequence$\left\{\gamma_{m}\right\}$. 3. (c)

By definition, for large enough $m$ we have $\gamma_{m}\leq 1$. Therefore, we have

[TABLE]

Next, we conclude by showing that Assumption C.1.4 is satisfied. Define the process $d_{n}\triangleq\sum_{m=0}^{n}c_{m}$. This process is a martingale since the sequence $\left\{c_{m}\right\}$ is square integrable (by Lemma C.9) and satisfies $\textrm{E}\left[d_{m+1}|d_{m}\right]=d_{m}$. Using Doob’s martingale inequality111If $w_{n}$ is a martingale sequence then $\Pr\left(\sup_{m\geq 0}\left|w_{n}\right|\geq\mu\right)\leq\lim_{n\rightarrow\infty}\textrm{E}\left[\left|w_{n}\right|^{2}\right]/\mu^{2}$. we have

[TABLE]

C.5 Satisfying Assumption C.1.5

In this section we need to show that for each $\mu>0$ and for some $T>0$ we have

[TABLE]

In order to follow the same lines as in Section C.4, we need to show that the second moment of the martingale difference noise, $\delta M_{i}$, is bounded with zero mean. By definition, $\delta M_{n}\left(\cdot\right)$ has zero mean.

Lemma C.10

The martingale difference noise, $\delta M_{n}\left(\cdot\right)$, is bounded in the second moment.

**Proof ** The claim is immediate from the fact that

[TABLE]

and from Lemma C.3, Lemma C.4, and Lemma C.5.  

Combining this fact with Lemma C.10, and applying the regenerative decomposition of Section C.4, we conclude that statistically $\delta M_{n}\left(\cdot\right)$ behaves exactly as $\left(g_{n}\left(y,x_{i}\right)-\bar{g}\left(y\right)\right)$ of section C.4 and thus (44) holds.

C.6 Satisfying Assumption C.1.6

In this section we need to prove that there are non-negative measurable functions $\rho_{3}\left(y\right)$ and $\rho_{n4}\left(x\right)$ such that

[TABLE]

where $\rho_{3}\left(y\right)$ is bounded on each bounded $y$-set, and for each $\mu>0$ we have

[TABLE]

The following lemma states a stronger condition for Assumption C.1.6. In fact, we choose $\rho_{3}(y)$ to be a positive constant.

Lemma C.11

If $\left\|g_{n}\left(y,x\right)\right\|$ is uniformly bounded for each $y$, $x$ and $n$, then Assumption C.1.6 is satisfied.

**Proof ** Let us denote the upper bound by the random variable $B$, i.e.,

[TABLE]

Thus

[TABLE]

Based on Lemma C.11, we are left with proving that $g_{n}\left(y,x\right)$ is uniformly bounded. The following lemma states so.

Lemma C.12

The function $g_{n}\left(y,x\right)$ is* uniformly bounded for all $n$.*

**Proof ** We examine the components of $g_{n}\left(y_{n},x_{n}\right)$. In (C.1) we showed that

[TABLE]

Since both $r\left(x_{n}\right)$ and $\tilde{\eta}_{n}$ are bounded by Assumption 3.1 and Lemma C.3 respectively, we have a uniform bound on $g_{n}\left(\tilde{\eta}_{n},x_{n}\right)$. Recalling (C.1) we have

[TABLE]

Finally, recalling (C.1) we have

[TABLE]

C.7 Satisfying Assumption C.1.7

In this section we show that there are non-negative measurable functions $\rho_{1}\left(y\right)$ and $\rho_{n2}\left(x\right)$ such that $\rho_{1}\left(y\right)$ is bounded on each bounded $y$-set and

[TABLE]

where

[TABLE]

and for some $\tau>0$

[TABLE]

From Section C.6 we infer that we can choose $\rho_{n2}\left(x\right)$ to be a constant since $g_{n}\left(y,x\right)$ is uniformly bounded. Thus, we need to show the appropriate $\rho_{1}\left(\cdot\right)$ function. The following lemma shows it.

Lemma C.13

The following functions satisfy (46) and (47).

The function $\rho_{1}\left(y\right)=\left\|\tilde{\eta}_{2}-\tilde{\eta}_{1}\right\|$ and $\rho_{n2}\left(x\right)=\Gamma_{\eta}$ for $g_{n}\left(\tilde{\eta},x\right)$.

1. (a)

The function $\rho_{1}\left(y\right)=\frac{1}{1-\lambda}B_{\phi}^{2}\left(\sum_{y\in\mathcal{X}}B_{w}\left\|P\left(y|x,\theta_{1}\right)-P\left(y|x,\theta_{2}\right)\right\|+\left\|w_{1}-w_{2}\right\|\right)$ and $\rho_{n2}\left(x\right)=\Gamma_{w}$ for $g_{n}\left(w,x\right)$. 2. (b)

The function $\rho_{1}\left(y\right)=\sum_{y\in\mathcal{X}}B_{w}\left\|P\left(y|x,\theta_{1}\right)-P\left(y|x,\theta_{2}\right)\right\|\cdot B_{\psi}$ and $\rho_{n2}\left(x\right)=1$ for $g_{n}\left(\theta,x\right)$.

**Proof **

Recalling (C.1) we have for $g_{n}\left(\tilde{\eta},x\right)$

[TABLE]

thus (46) and (47) are satisfied for 1. 2. 2.

Recalling (C.1) we have for $g_{n}\left(w,x\right)$

[TABLE]

1. (a)

Trivially, with respect to $w$ (46) and (47) are satisfied. Regarding $\theta$, (46) and (47) are satisfied if we recall the definition of $P\left(y|x,\theta\right)$ from (1) and the continuity of $\mu\left(u|x,\theta\right)$ from Assumption 3.4. 2. (b)

Recalling (C.1) we have for $g_{n}\left(\theta,x\right)$

[TABLE]

Using similar arguments to 2, (46) and (47) are satisfied for $\theta$.

D Proof of Theorem 4.7

In this section we find conditions under which Algorithm 1 converges to a neighborhood of a local maximum. More precisely, we show that $\liminf_{t\rightarrow\infty}\left\|\nabla\eta(\theta(t))\right\|_{2}\leq\epsilon_{\textrm{app}}+\epsilon_{\textrm{dyn}}$, where the approximation error, $\epsilon_{\textrm{app}}$, measures the error inherent in the critic’s representation, and $\epsilon_{\textrm{dyn}}$ is an error related to the single time scale algorithm. We note that the approximation error depends on the basis functions chosen for the critic, and in general can be reduced only by choosing a better representation basis. The term $\epsilon_{\textrm{dyn}}$ is the dynamic error, and this error can be reduced by choosing the critic’s parameters $\Gamma_{\eta}$ and $\Gamma_{w}$ appropriately.

We begin by establishing a variant of Lyapunov’s theorem for asymptotic stability222We say that the equilibrium point $x=0$ of the system $\dot{x}$=f$\left(x\right)$ is stable if for each $\epsilon>0$ there exists a $\delta>0$ such that $\left\|x\left(0\right)\right\|<\delta\Rightarrow\left\|x\left(t\right)\right\|<\epsilon$ for all $t\geq 0$. We say that the point $x=0$ is asymptotically stable if it is stable and there exists a $\delta>0$ such that $\left\|x\left(0\right)\right\|<\delta$ implies $\lim_{t\rightarrow\infty}x\left(t\right)=0$ (see Khalil (2002) for more details). , where instead of proving asymptotic convergence to a point, we prove convergence to a compact invariant set. Based on this result, we continue by establishing a bound on a time dependent ODE of the first order. This result is used to bound the critic’s error in estimating the average reward per stage and the differential values. Finally, using these results, we establish Theorem 4.7.

We denote a closed ball of radius $y$ in some normed vector space, $(\mathbb{R}^{L},\|\cdot\|_{2})$, by $\mathcal{B}_{y}$, and its surface by $\partial\mathcal{B}_{y}$. Also, we denote by $A\backslash B$ a set, which contains all the members of set $A$ which are not members of $B$. Finally, we define the complement of $\mathcal{B}_{y}$ by $\mathcal{B}_{y}^{c}=\mathbb{R}^{L}\backslash\mathcal{B}_{y}$.

The following lemma is similar to Lyapunov’s classic theorem for asymptotic stability (Khalil (2002), Theorem 4.1). The main difference is that when the value of the Lyapunov function is unknown inside a ball, convergence can be established to the ball, rather than to a single point.

Lemma D.1

Consider a dynamical system, $\dot{x}=f\left(x\right)$ in a normed vector space, $(\mathbb{R}^{L},\|\cdot\|)$, and a closed ball $\mathcal{B}_{r}\triangleq\left\{x\left|x\in\mathbb{R}^{L},\|x\|\leq r\right.\right\}$. Suppose that there exists a continuously differentiable scalar function $V\left(x\right)$ such that $V\left(x\right)>0$ and $\dot{V}\left(x\right)<0$ for all $x\in\mathcal{B}_{r}^{c}$, and $V\left(x\right)=0$ for $x\in\partial\mathcal{B}_{r}$. Then,

[TABLE]

**Proof ** We prove two complementary cases. In the first case, we assume that $x\left(t\right)$ never enters $\mathcal{B}_{r}$. On the set $\mathcal{B}_{r}^{c}$, $V\left(x\right)$ is a strictly decreasing function in $t$, and it is bounded below, thus it converges. We denote this bound by $C$, and notice that $C\geq 0$ since for $x\in\mathcal{B}_{r}^{c}$, $V\left(x\right)>0$. We prove that $C=0$ by contradiction. Assume that $C>0$. Then, $x(t)$ converge to the invariant set $S_{C}\triangleq\{x|V\left(x\right)=C,x\in\mathcal{B}_{r}^{c}\}$. For each $x\left(t\right)\in\mathcal{S}_{C}$ we have $\dot{V}\left(x\right)<0$. Thus, $V\left(x\right)$ continues to decrease which contradicts the boundedness from below. As a result, $V(x\left(t\right))\rightarrow 0$.

In the second case, let us suppose that at some time, denoted by $t_{0}$, $x(t_{0})\in\mathcal{B}_{r}$. We argue that the trajectory never leaves $\mathcal{B}_{r}$. Let us assume that at some time $t_{2}$, the trajectory $x(t)$ enters the set $\partial\mathcal{B}_{r+\epsilon}$. Then on this set, we have $V(x(t_{2}))>0$. By the continuity of the trajectory $x(t)$, the trajectory must go through the set $\partial\mathcal{B}_{r}$. Denote the hitting time of this set by $t_{1}$. By definition we have $V(x(t_{1}))=0$. Without loss of generality, we assume that the trajectory in the times $t_{1}<t\leq t_{2}$ is restricted to the set $\mathcal{B}_{r+\epsilon}/\mathcal{B}_{r}$. Thus, since $\dot{V}(x(t))\leq 0$ for $x\in\mathcal{B}_{r+\epsilon}/\mathcal{B}_{r}$ we have

[TABLE]

which contradicts the fact that $V(x(t_{2}))\geq V(x(t_{1}))$. Since this argument holds for all $\epsilon>0$, the trajectory $x(t)$ never leaves $\mathcal{B}_{r}$.  

The following lemma will be applied later to the linear equations (27), and more specifically, to the ODEs describing the dynamics of $\tilde{\eta}$ and $w$. It bounds the difference between an ODE’s state variables and some time dependent functions.

Lemma D.2

Consider the following ODE in a normed space $(\mathbb{R}^{L},\|\cdot\|_{2})$

[TABLE]

where for sufficiently large $t$ .

$\mathcal{M}(t)\in\mathbb{R}^{L\times L}$* is a continuous matrix which satisfies $\max_{\left\|x\right\|=1}x^{\prime}\mathcal{M}\left(t\right)x\leq-\gamma<0$ for $t\in\mathbb{R}$,* 2. 2.

$F_{1}\left(t\right)\in\mathbb{R}^{L}$* satisfies $\|dF_{1}(t)/dt\|_{2}\leq B_{F1}$,* 3. 3.

$F_{2}\left(t\right)\in\mathbb{R}^{L}$* satisfies $\|F_{2}(t)\|_{2}\leq B_{F2}$.*

Then, the solution of the ODE satisfies $\limsup_{t\rightarrow 0}\|X(t)-F_{1}\left(t\right)\|_{2}\leq\left(B_{F1}+B_{F2}\right)/\gamma$.

**Proof ** We express (48) as

[TABLE]

and define

[TABLE]

Therefore, (49) can be written as

[TABLE]

where $\|G(t)\|\leq B_{G}\triangleq B_{F1}+B_{F2}$. In view of Lemma D.1, we consider the function

[TABLE]

Let $\mathcal{B}_{r}$ be a ball with a radius $r=B_{G}/\gamma$. Thus we have $V\left(Z\right)>0$ for $Z\in\mathcal{B}_{r}^{c}$ and $V(Z)=0$ for $X\in\partial\mathcal{B}_{r}$. In order to satisfy the assumptions of Lemma D.1 the condition that $\dot{V}(Z)<0$ needs to be verified. For $\left\|Z(t)\right\|_{2}>B_{G}/\gamma$ we have

[TABLE]

As a result, the assumptions of Lemma D.1 are valid and the Lemma is proved.  

The following lemma shows that the matrix $A\left(\theta\right)$, defined in (26), satisfies the conditions of Lemma D.2. For the following lemmas, we define the weighted norm $\left\|w\right\|_{\Pi\left(\theta\right)}^{2}\triangleq\left\|w^{\prime}\Pi\left(\theta\right)w\right\|_{2}$.

Lemma D.3

The following inequalities hold:

For any $w\in\mathbb{R}^{L}$and for all $\theta\in\mathbb{R}^{K}$, $\left\|P\left(\theta\right)w\right\|_{\Pi\left(\theta\right)}<\left\|w\right\|_{\Pi\left(\theta\right)}$..

1. (a)

The matrix $M\left(\theta\right)$ satisfies $\left\|M\left(\theta\right)w\right\|_{\Pi\left(\theta\right)}<\left\|w\right\|_{\Pi\left(\theta\right)}$ for all $\theta\in\mathbb{R}^{K}$ and $w\in\mathbb{R}^{L}$. 2. (b)

The matrix $\Pi\left(\theta\right)\left(M\left(\theta\right)-I\right)$ satisfies $x^{\prime}\Pi\left(\theta\right)\left(M\left(\theta\right)-I\right)x<0$ for all $x\in\mathbb{R}^{L}$ and for all $\theta\in\mathbb{R}^{K}$. 3. (c)

There exists a positive scalar $\gamma$ such that $w^{\prime}A\left(\theta\right)w<-\gamma$ for all $w^{\prime}w=1$.

**Proof ** The following proof is similar in many aspects to the proof of Lemma 6.6 of Bertsekas and Tsitsiklis (1996).

By using Jensen’s inequality for the function $f\left(\alpha\right)=\alpha^{2}$ we have

[TABLE]

If in Jensen’s inequality we have a strictly convex fiction and non-degenerate probability measures then the inequality is strict. The function $f\left(\alpha\right)$ is strictly convex, and by Assumption 3.2 the matrix $P\left(\theta\right)$ is aperiodic, which implies that the matrix $P\left(\theta\right)$ is not a permutation matrix. As a result, there exists $x_{0}\in\mathcal{X}$ such that the probability measure $P\left(y|x_{0},\theta\right)$ is not degenerate, thus, the inequality in (50) is strict, i.e.,

[TABLE]

Then, we have

[TABLE]

where in the inequality we have used (51).

1. (a)

Using the triangle inequality and 1 we have

[TABLE] 2. (b)

By definition

[TABLE]

where in the first inequality we have used the Cauchy-Schwartz inequality, and in the second inequality we have used 1. Thus, $x^{\prime}\Pi\left(\theta\right)\left(M\left(\theta\right)-I\right)x<0$ for all $x\in\mathbb{R},$ which implies that $\Pi\left(\theta\right)\left(M\left(\theta\right)-I\right)$ is a negative definite (ND) matrix333Usually, a ND matrix is defined for Hermitian matrices, i.e., if $B$ is an Hermitian matrix and it satisfies $x^{\prime}Bx<0$ for all $x\in\mathbb{C}^{K}$ then $B$ is a NSD matrix . We use here a different definition which states that a square matrix $B$ is a ND matrix if it is real and it satisfies $x^{\prime}Bx<0$ for all $x\in\mathbb{R}^{k}$ (see Horn and Johnson (1985) p. 399).. 3. (c)

From 3, we know that for all $\theta\in\mathbb{R}^{K}$ and all $w\in\mathbb{R}^{\left|\mathcal{X}\right|}$ satisfying $w^{\prime}w=1$, we have $w^{\prime}\Pi\left(\theta\right)\left(M\left(\theta\right)-I\right)w<0$, and by Assumption (3.2), this is true also for the closure of $\left\{\Pi\left(\theta\right)\left(M\left(\theta\right)-I\right)|\theta\in\mathbb{R}^{K}\right\}$. Thus, there exists a positive scalar, $\gamma^{\prime}$, satisfying

[TABLE]

By Assumption 4.3 the rank of the matrix $\Phi$ is full, thus there exists a scalar $\gamma$ such that for all $w\in\mathbb{R}^{L}$, where $w^{\prime}w=1$, we have $w^{\prime}A\left(\theta\right)w\leq-\gamma<0$.

The following Lemma establishes the boundedness of $\dot{\theta}$.

Lemma D.4

There exists a constant $B_{\theta 1}\triangleq B_{\eta 1}+B_{\psi}\left(B_{D}+B_{r}+B_{\tilde{\eta}}+2B_{\phi}B_{w}\right)$ such that $\|\dot{\theta}\|_{2}\leq B_{\theta 1}$.

**Proof ** Recalling (27)

[TABLE]

Based on Lemma (D.4), the following Lemma shows the boundedness of $\left(\eta(\theta(t))-\tilde{\eta}\right)$.

Lemma D.5

We have

[TABLE]

where $B_{\Delta\eta}\triangleq B_{\eta_{1}}B_{\theta 1}$.

**Proof ** Using the Cauchy-Schwartz inequality we have

[TABLE]

Recalling the equation for $\tilde{\eta}$ in (27) we have

[TABLE]

We conclude by applying Lemma D.2 and using (52) that

[TABLE]

In (53) we see that the bound on $|\eta(\theta)-\tilde{\eta}|$ is controlled by $\Gamma_{\eta}$, where larger values of $\Gamma_{\eta}$ ensure smaller values of $|\eta(\theta)-\tilde{\eta}|$. Next, we bound $\|w^{*}(\theta)-w\|_{2}$. We recall the second equation of (27)

[TABLE]

We can write the equation for $\dot{w}$ as

[TABLE]

where $w^{*}=-A\left(\theta\right)^{-1}b\left(\theta\right)$. In order to use Lemma D.2, we need to demonstrate the boundedness of $\left\|\frac{d}{dt}w^{*}\right\|$. The following lemma does so.

Lemma D.6

There exists a positive constant, $B_{b}\triangleq\frac{1}{1-\lambda}\left|\mathcal{X}\right|^{3}LB_{\Phi}B_{r}$, such that $\left\|b\left(\theta\right)\right\|_{2}\leq B_{b}$.

1. (a)

There exists a positive constant, $B_{G}\triangleq\frac{1}{1-\lambda}\left|\mathcal{X}\right|^{3}LB_{\Phi}$, such that $\left\|G\left(\theta\right)\right\|_{2}\leq B_{G}$. 2. (b)

There exist positive constants, $\tilde{B}=B_{\pi 1}\left(B_{r}+B_{\eta}\right)B_{\theta 1}+B_{P1}\left(B_{r}+B_{\eta}\right)B_{\theta 1}+B_{\eta 1}B_{\theta 1}$ and $B_{b1}\triangleq\frac{1}{1-\lambda}\left|\mathcal{X}\right|^{3}B_{\Phi}B_{r}\tilde{B}$, such that we have $\left\|\dot{b}\left(\theta\right)\right\|_{2}\leq B_{b1}.$ 3. (c)

There exist constants $b_{A}$ and $B_{A}$ such that

[TABLE] 4. (d)

There exist a constants $B_{A1}$ such that

[TABLE] 5. (e)

We have

[TABLE] 6. (f)

There exists a positive constant, $B_{w1}$, such that

[TABLE]

**Proof **

We show that the entries of the vector $b\left(\theta\right)$ are uniformly bounded in $\theta$, therefore, its norm is uniformly bounded in $\theta$. Let us look at the $i$-th entry of the vector $b\left(\theta\right)$ (we denote by $\left[\cdot\right]_{j}$ the $j$-th row of a matrix or a vector)

[TABLE]

thus $\left\|b\left(\theta\right)\right\|_{2}\leq\frac{1}{1-\lambda}\left|\mathcal{X}\right|^{3}LB_{\Phi}B_{r}$ is uniformly bounded in $\theta$. 2. 2.

The proof is accomplished by similar argument to section 1. 3. 3.

Similarly to section 1, we show that the entries of the vector $\dot{b}\left(\theta\right)$ are uniformly bounded in $\theta$, therefore, its norm is uniformly bounded in $\theta$. First, we show that the following function of $\theta\left(t\right)$ is bounded.

[TABLE]

where we used the triangle and Cauchy-Schwartz inequalities in the first and second inequalities respectively, and Lemmas 3.6 and D.4 in the second inequality. Thus,

[TABLE] 4. 4.

Since $A(\theta)$ satisfies $y^{\prime}A(\theta)y<0$ for all nonzero $y$, it follows that all its eigenvalues are nonzero. Therefore, the eigenvalues of $A\left(\theta\right)^{\prime}A\left(\theta\right)$ are all positive and real since $A\left(\theta\right)^{\prime}A\left(\theta\right)$ is a symmetric matrix. Since by Assumption 3.2 this holds for all $\theta\in\mathbb{R}^{K}$, there is a global minimum, $b_{A}$, and a global maximum, $B_{A}$, such that

[TABLE]

where we denote by $\lambda_{\textrm{min}}\left(\cdot\right)$ and $\lambda_{\textrm{max}}\left(\cdot\right)$ the minimal and maximal eigenvalues of the matrix respectively. Using Horn and Johnson (1985) section 5.6.6, we have $\lambda_{max}\left(A\left(\theta\right)^{\prime}A\left(\theta\right)\right)=\left\|A\left(\theta\right)\right\|_{2}$, thus, we get an upper bound on the matrix norm. Let us look at the norm of $\left\|A\left(\theta\right)^{-1}\right\|_{2}$,

[TABLE]

thus, we the lower bound on $\left\|A\left(\theta\right)^{-1}\right\|_{2}$ is $\sqrt{1/\lambda_{\mathrm{min}}\left(A\left(\theta\right)^{\prime}A\left(\theta\right)\right)}$, i.e., $b_{A}$. 5. 5.

Let us look at the $ij$ entry of the matrix $\frac{d}{dt}A\left(\theta\right)$, where using similar arguments to section 2 we get

[TABLE]

Since the matrix entries are uniformly bounded in $\theta$, so is the matrix $\frac{d}{dt}A\left(\theta\right)^{\prime}\frac{d}{dt}A\left(\theta\right)$, and so is the largest eigenvalue of $\frac{d}{dt}A\left(\theta\right)^{\prime}\frac{d}{dt}A\left(\theta\right)$ which implies the uniform boundedness of $\left\|\frac{d}{dt}A\left(\theta\right)\right\|_{2}$. 6. 6.

For a general invertible square matrix, $X\left(t\right)$, we have

[TABLE]

Rearranging it we get

[TABLE]

Using this identity yields

[TABLE] 7. 7.

Examining the norm of $\frac{d}{dt}w^{*}$ yields

[TABLE]

We wish to use Lemma D.2 for (27), thus, we show that the assumptions of Lemma D.2 are valid.

Lemma D.7

We have

[TABLE]

where

[TABLE]

1. (a)

We have

[TABLE]

where

[TABLE]

**Proof **

Without loss of generality, we can eliminate the projection operator since we can choose $B_{w}$ to be large enough such that $w^{*}(\theta)$ will be inside the bounded space. We take $\mathcal{M}(t)=A\left(\theta\right)$, $F_{1}\left(t\right)=w^{*}(\theta(t))$, and $F_{2}\left(t\right)=G(\theta)(\eta(\theta)-\tilde{\eta})$ . By previous lemmas we can see that the Assumption D.2 holds. By Lemma D.6 (6), $\left\|\dot{w}^{*}(\theta)\right\|_{2}$ is bounded by $B_{w_{1}}$, by Lemma D.5 we have a bound on $|(\eta(\theta)-\tilde{\eta})|$, and by Lemma D.3 we have a bound on $w^{\prime}A\left(\theta\right)w$. Using these bounds and applying Lemma D.2 provides the desired result.

1. (a)

Suppressing the time dependence for simplicity and expressing $\|h(\theta)-\tilde{h}(w)\|_{\infty}$ using $\epsilon_{\textrm{app}}$ and the previous result yields

[TABLE]

For the first term on the r.h.s. of the final equation in (55) we have

[TABLE]

where we use the sub-additivity of the matrix norms in the first inequality, and Lemma 3.6 and the (17) in the last inequality. For the second term on the r.h.s.​ of the final equation in (55) we have

[TABLE]

Combining (54)-(57) yields the desired result.

Using Lemma D.7 we can provide a bound on second term of (27).

Lemma D.8

We have

[TABLE]

where

[TABLE]

**Proof ** Simplifying the notation by suppressing the time dependence, we bound the TD signal in the limit, i.e.,

[TABLE]

With some more algebra we have

[TABLE]

We see that the term in this bound is adjustable by choosing appropriate $\Gamma_{\eta}$ and $\Gamma_{w}$. The concluding lemma proves the conclusion of Theorem 4.7.

Proof of Theorem 4.7

We define

[TABLE]

For an arbitrary $\delta>0$, define the set

[TABLE]

We claim that the trajectory $\eta(\theta)$ visits $\mathcal{B}_{\delta}$ infinitely often. Assume the contrary that

[TABLE]

Thus, on the set $\mathcal{B}_{\delta}^{c}$ for $t$ large enough we have

[TABLE]

By (59), there exists a time $t_{0}$ which for all $t>t_{0}$ we have $\eta(\theta)\in\mathcal{B}_{\delta}^{c}$. Therefore,

[TABLE]

which contradicts the boundedness of $\eta(\theta)$. Since the claim holds for all $\delta>0$, the result follows.