On Generalized Bellman Equations and Temporal-Difference Learning
Huizhen Yu, A. Rupam Mahmood, Richard S. Sutton
TL;DR
This paper introduces a flexible scheme for setting the $\lambda$-parameters in off-policy TD learning using generalized Bellman equations, improving stability and allowing larger $\lambda$ values, with proven ergodicity and convergence properties.
Contribution
It proposes a novel, more direct method for setting $\\lambda$ in off-policy TD learning based on eligibility traces, enhancing stability and flexibility.
Findings
The scheme maintains bounded traces in off-policy TD learning.
It establishes ergodicity of the joint state-trace process.
It characterizes the convergence behavior of least-squares implementations.
Abstract
We consider off-policy temporal-difference (TD) learning in discounted Markov decision processes, where the goal is to evaluate a policy in a model-free way by using observations of a state process generated without executing the policy. To curb the high variance issue in off-policy TD learning, we propose a new scheme of setting the -parameters of TD, based on generalized Bellman equations. Our scheme is to set according to the eligibility trace iterates calculated in TD, thereby easily keeping these traces in a desired bounded range. Compared with prior work, this scheme is more direct and flexible, and allows much larger values for off-policy TD learning with bounded traces. As to its soundness, using Markov chain theory, we prove the ergodicity of the joint state-trace process under nonrestrictive conditions, and we show that associated with our scheme…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsReinforcement Learning in Robotics · Machine Learning and Algorithms · Markov Chains and Monte Carlo Methods
On Generalized Bellman Equations and
Temporal-Difference Learning††thanks: An abridged version of this article appeared at The th Canadian Conference on Artificial Intelligence (CAI), May 2017, and a reformatted copy is to appear in Journal on Machine Learning Research, 2018.
Huizhen Yu
RLAI Lab, Department of Computing Science
University of Alberta, Canada
A. Rupam Mahmood
Kindred Inc.
243 College St
Toronto, ON M5T 1R5, Canada
Richard S. Sutton
RLAI Lab, Department of Computing Science
University of Alberta, Canada
Abstract
We consider off-policy temporal-difference (TD) learning in discounted Markov decision processes, where the goal is to evaluate a policy in a model-free way by using observations of a state process generated without executing the policy. To curb the high variance issue in off-policy TD learning, we propose a new scheme of setting the -parameters of TD, based on generalized Bellman equations. Our scheme is to set according to the eligibility trace iterates calculated in TD, thereby easily keeping these traces in a desired bounded range. Compared with prior work, this scheme is more direct and flexible, and allows much larger values for off-policy TD learning with bounded traces. As to its soundness, using Markov chain theory, we prove the ergodicity of the joint state-trace process under nonrestrictive conditions, and we show that associated with our scheme is a generalized Bellman equation (for the policy to be evaluated) that depends on both the evolution of and the unique invariant probability measure of the state-trace process. These results not only lead immediately to a characterization of the convergence behavior of least-squares based implementation of our scheme, but also prepare the ground for further analysis of gradient-based implementations.
Keywords: Markov decision process; approximate policy evaluation; generalized Bellman equation; reinforcement learning; temporal-difference method; Markov chain; randomized stopping time
Contents
-
3.1 Randomized Stopping Times and Associated Bellman Operators
-
Appendix B: Oblique Projection Viewpoint and Error Bound for TD
-
B.1 Solutions of TD as Oblique Projections of the Value Function
1 Introduction
We consider discounted Markov decision processes (MDPs) and off-policy temporal-difference (TD) learning methods for approximate policy evaluation with linear function approximation. The goal is to evaluate a policy in a model-free way by using observations of a state process generated without executing the policy. Off-policy learning is an important part of the reinforcement learning methodology (Sutton and Barto, 1998) and has been studied in the areas of operations research and machine learning. (For an incomplete list of references, see e.g., Glynn and Iglehart, 1989; Precup et al., 2000, 2001; Randhawa and Juneja, 2004; Sutton et al., 2008, 2009; Maei, 2011; Yu, 2012; Dann et al., 2014; Geist and Scherrer, 2014; Mahadevan et al., 2014; Mahmood et al., 2014; Liu et al., 2015; Sutton et al., 2016; Dai et al., 2018.) Available TD algorithms, however, tend to have very high variances due to the use of importance sampling, an issue that limits their applicability in practice. The purpose of this paper is to introduce a new TD learning scheme that can help address this problem.
Our work is motivated by the recently proposed Retrace algorithm (Munos et al., 2016) and ABQ algorithm (Mahmood et al., 2017), and by the Tree-Backup algorithm (Precup et al., 2000) that existed earlier. These algorithms, as explained by Mahmood et al. (2017), all try to use the -parameters of TD to curb the high variance issue in off-policy learning. In particular, they all choose the values of according to the current state or state-action pair in such a way that guarantees the boundedness of the eligibility traces in TD learning, which can help reduce significantly the variance of the TD iterates. A limitation of these algorithms, however, is that they tend to be over-conservative and restrict to small values, whereas small can result in large approximation bias in TD solutions.
In this paper, we propose a new scheme of setting the -parameters of TD, based on generalized Bellman equations. Our scheme is to set according to the eligibility trace iterates calculated in TD, thereby easily keeping those traces in a desired bounded range. Compared with the schemes used in the previous work just mentioned, this is a direct way to bound the traces in TD, and it is also more flexible and allows much larger values for off-policy learning.
Regarding generalized Bellman equations, in our context, they will correspond to a family of dynamic programming equations for the policy to be evaluated. These equations all have the true value function as their unique solution, and their associated operators have contraction properties, like the standard Bellman operator. We will refer to the associated operators as generalized Bellman operators or Bellman operators for short. Some authors have considered, at least conceptually, the use of an even broader class of equations for policy evaluation. For example, Ueno et al. (2011) have considered treating the policy evaluation problem as a parameter estimation problem in the statistical framework of estimating equations, and in their framework, any equation that has the true value function as the unique solution can be used to estimate the value function. The family of generalized Bellman equations we consider has a more specific structure. They generalize multistep Bellman equations, and they are associated with randomized stopping times and arise from the strong Markov property (see Section 3.1 for details).
Generalized Bellman equations and operators are powerful tools. In classic MDP theory they have been used in some intricate optimality analyses (e.g., Schäl and Sudderth, 1987). Their computational use, however, seems to emerge primarily in the field of reinforcement learning. Through the -parameters and eligibility traces, TD learning is naturally connected with, not a single Bellman operator, but a family of Bellman operators, with different choices of or different rules of calculating the eligibility trace iterates corresponding to different Bellman operators. Early efforts that use this aspect to broaden the scope of TD algorithms and to analyze such algorithms include Sutton’s work (1995) on learning at multiple timescales and Tsitsiklis’ work on generalized TD algorithms in the tabular case (see the book by Bertsekas and Tsitsiklis, 1996, Chap. 5.3). In the context of off-policy learning, there are more recent approaches that try to utilize this connection of TD with generalized Bellman operators to make TD learning more efficient (Precup et al., 2000; Yu and Bertsekas, 2012; Munos et al., 2016; Mahmood et al., 2017). This is also our aim, in proposing the new scheme of setting the -parameters.
Our analyses of the new TD learning scheme will focus on its theoretical side. Using Markov chain theory, we prove the ergodicity of the joint state and trace process under nonrestrictive conditions (see Theorem 2.1), and we show that associated with our scheme is a generalized Bellman equation (for the policy to be evaluated) that depends on both the evolution of and the unique invariant probability measure of the state-trace process (see Theorem 3.2 and Corollary 3.1). These results not only lead immediately to a characterization of the convergence behavior of least-squares based implementation of our scheme (see Corollary 2.1 and Remark 3.2), but also prepare the ground for further analysis of gradient-based implementations. (The latter analysis has been carried out recently by Yu (2017); see Remark 3.3.)
In addition to the theoretical study, we also present the results from a preliminary numerical study that compares several ways of setting for the least-squares based off-policy algorithm. The results demonstrate the advantages of the proposed new scheme with its greater flexibility.
We remark that although we focus exclusively on policy evaluation in this paper, approximate policy evaluation methods are highly pertinent to finding near-optimal policies in MDPs. They can be applied in approximate policy iteration, in policy-gradient algorithms for gradient estimation or in direct policy search (see e.g., Konda, 2002; Mannor et al., 2003). In addition to solving MDPs, they can also be used in artificial intelligence and robotics applications as a means to generate experience-based world models (see e.g., Sutton, 2009). It is, however, beyond the scope of this paper to discuss these applications of our results.
The rest of the paper is organized as follows. In Section 2, after a brief background introduction, we present our scheme of TD learning with bounded traces, and we establish the ergodicity of the joint state-trace process. In Section 3, we first discuss generalized Bellman operators associated with randomized stopping times, and we then derive the generalized Bellman equation associated with our scheme. In Section 4, we present the experimental results on the least-squares based implementation of our scheme. Appendices Appendix A: Proof of Theorem 3.1-Appendix B: Oblique Projection Viewpoint and Error Bound for TD include a proof for generalized Bellman operators and materials about approximation properties of TD solutions that are too long to include in the main text.
2 Off-Policy TD Learning with Bounded Traces
We describe the off-policy policy evaluation problem and the algorithmic form of TD learning in Section 2.1. We then present our scheme of history-dependent in Section 2.2, and analyze the properties of the resulting eligibility trace iterates and the convergence of the corresponding least-squares based algorithm in Section 2.3.
2.1 Preliminaries
The off-policy learning problem we consider in this paper concerns two Markov chains on a finite state space . The first chain has transition matrix , and the second . Whatever physical mechanisms that induce the two chains shall be denoted by and , and referred to as the target policy and behavior policy, respectively. The second Markov chain we can observe; however, it is the system performance of the first Markov chain that we want to evaluate.
Specifically, we consider a one-stage reward function and an associated discounted total reward criterion with state-dependent discount factors . Let denote the diagonal matrix with diagonal entries . We assume that and satisfy the following conditions:
Condition 2.1** (Conditions on the target and behavior policies).**
- (i)
* is such that the inverse exists, and*
- (ii)
* is such that for all , , and moreover, is irreducible.*
The performance of is defined as the expected discounted total rewards for each initial state :
[TABLE]
where the notation means that the expectation is taken with respect to (w.r.t.) the Markov chain starting from and induced by (i.e., with transition matrix ). The function is well-defined under Condition 2.1(i). It is called the value function of , and by standard MDP theory (see e.g., Puterman, 1994), we can write it in matrix/vector notation as
[TABLE]
The first equation above is known as the Bellman equation (or dynamic programming equation) for a stationary policy (cf. Footnote 2).
We compute an approximation of of the form , , where is a parameter vector and is an -dimensional feature representation for each state (here are column vectors and the symbol stands for transpose). Data available for this computation are:
- (i)
a realization of the Markov chain with transition matrix generated by , and
- (ii)
rewards associated with state transitions, where the function relates to as for all .111One can add to a zero-mean finite-variance noise term. This makes little difference to our analyses, so we have left it out for notational simplicity.
To find a suitable parameter for the approximation , we use the off-policy TD learning scheme. Define (the importance sampling ratio),222Our problem formulation entails both value function and state-action value function estimation for a stationary policy in the standard MDP context. In these applications, it is the state-action space of the MDP that corresponds to the state space here. In particular, for value function estimation, here corresponds to the pair of previous action and current state in the MDP, whereas for state-value function estimation, here corresponds to the current state-action pair in the MDP. The ratio then comes out as the ratio of action probabilities under and , the same as what appears in most of the off-policy learning literature. For the details of these correspondences, see (Yu, 2012, Examples 2.1, 2.2). The third application is in a simulation context where corresponds to a simulated system and both are known so that the ratio is available. Such simulations are useful, for example, in studying system performance under perturbations, and in speeding up the computation when assessing the impacts of events that are rare under the dynamics . and write
[TABLE]
Given an initial , for each , the eligibility trace vector and the scalar temporal-difference term for any approximate value function are calculated according to
[TABLE]
Here , are important parameters in TD learning, the choice of which we shall elaborate on shortly.
There exist a number of TD algorithms that use and to generate a sequence of parameters for approximate value functions. One such algorithm is LSTD (Boyan, 1999; Yu, 2012), which obtains by solving the linear equation for ,
[TABLE]
(if it admits a solution), where is a matrix with row vectors . LSTD updates the equation (2.4) iteratively by incorporating one by one the observation of at each state transition. We will discuss primarily this algorithm in the paper, as its behavior can be characterized directly using our subsequent analyses of the joint state-trace process.
As mentioned earlier, our analyses will also provide bases for analyzing other gradient-based TD algorithms (e.g., Sutton et al., 2008, 2009; Maei, 2011; Mahadevan et al., 2014) by using stochastic approximation theory (Kushner and Yin, 2003; Borkar, 2008; Karmakar and Bhatnagar, 2018). Because of the complexity of this subject, however, we will not delve into it in the present paper, and we refer the reader to the recent work (Yu, 2017) for details.
2.2 Our Scheme of History-dependent
We now come to the choices of in the trace iterates (2.2). For TD with function approximation, one often lets be a constant or a function of (Sutton, 1988; Tsitsiklis and Van Roy, 1997; Sutton and Barto, 1998). If neither the behavior policy nor the ’s are further constrained, can have unbounded variances and is also unbounded in many natural situations (see e.g., Yu, 2012, Section 3.1), and this makes off-policy TD learning challenging.333However, asymptotic convergence can still be ensured for several algorithms (Yu, 2012, 2015, 2016b), thanks partly to a powerful law of large numbers for stationary processes. If we let the behavior policy to be close enough to the target policy so that , then variance can be reduced, but it is not a satisfactory solution, for the applicability of off-policy learning would be seriously limited.
Without restricting the behavior policy, as mentioned earlier, the two recent papers (Munos et al., 2016; Mahmood et al., 2017), as well as the closely related early work by Precup et al. (2000), exploit state-dependent ’s to control variance. Their choices of are such that for all , so that the trace iterates are made bounded, which can help reduce the variance of the iterates.
Motivated by this prior work, our proposal is to set according to directly, so that we can keep in a desired range straightforwardly and at the same time, allow a much larger range of values for the -parameters. As a simple example, if we use to scale the vector to be within a ball with some given radius, then we keep always bounded.
In the rest of this paper, we shall focus on analyzing the iteration (2.2) with a particular choice of of the kind just mentioned. We want to be more general than the preceding simple example. However, since the dependence on the trace would make dependent on the entire past history , we also want to retain certain Markovian properties that are very useful for convergence analysis. This leads us to consider being a certain function of the previous trace and past states. More specifically, we will let be a function of the previous trace and a certain memory state that is a summary of the states observed so far. The formulation is as follows.
2.2.1 Formulation and Examples
We denote the memory state at time by . For simplicity, we assume that can only take values from a finite set , and its evolution is Markovian: for some given function . The joint process is then a simple finite-state Markov chain. Each is a function of the history and . We further require, besides the irreducibility of (cf. Condition 2.1(ii)), that
Condition 2.2** (Evolution of memory states).**
Under the behavior policy , the Markov chain on has a single recurrent class.
This recurrence condition is nonrestrictive: If the Markov chain has multiple recurrent classes, each recurrent class can be treated separately by using the same arguments we present in this paper. However, we remark that the finiteness assumption on is a simplification. We choose to work with finite mainly for the reason that with the traces lying in a continuous space, to study the joint state and trace process, we need to resort to properties of Markov chains on infinite spaces. With an infinite , we would need to introduce more technical conditions that are not essential to our analysis and can obscure our main arguments.
We thus let and evolve as
[TABLE]
where . We require the function to satisfy two conditions.
Condition 2.3** (Conditions for ).**
For some norm on , the following hold for each memory state :
- (i)
For any , .
- (ii)
For some constant , for all and all possible state transitions that can lead to the memory state .
In the above, the second condition is to restrict in a desired range (as it makes ). The first condition is about the continuity of the function in the trace variable for each memory state , and it plays a key role in the subsequent analysis, where we will use this condition to ensure that the traces and the states jointly form a Markov chain with appealing properties. We shall defer a further discussion on the technical roles of these conditions to the end of Section 2.3 (cf. Remark 2.2).
Let us give a few simple examples of choosing that satisfy Condition 2.3. We will later use these examples in our experimental study (Section 4).
Example 2.1**.**
We consider again the simple scaling example mentioned earlier and describe it using the terminologies just introduced. In this example, we let . For each , we define the function so that when multiplied with , the vector is scaled down whenever its length exceeds a given threshold :
[TABLE]
Condition 2.3(i) is satisfied because for with , \lambda\big{(}y,e\big{)}e=e, whereas for with , \lambda\big{(}y,e\big{)}e is simply the Euclidean projection of onto the ball (centered at the origin) with radius and is therefore Lipschitz continuous in with modulus w.r.t. . Corresponding to (2.6), the update rule (2.2) of becomes
[TABLE]
Note that this scheme of setting encourages the use of large : will be chosen whenever possible. A variation of the scheme is to multiply the right-hand side (r.h.s.) of (2.6) by another factor , so that can be at most . In particular, one such variation is to simply multiply the r.h.s. of (2.6) by a constant so that for all . ∎
Example 2.2**.**
The Retrace algorithm (Munos et al., 2016) modifies the trace updates in off-policy TD learning by truncating the importance sampling ratios by . In particular, for the off-policy TD() algorithm with a constant , Retrace modifies the trace updates to be
[TABLE]
As pointed out by Mahmood et al. (2017), to retain the original interpretation of as a bootstrapping parameter in TD learning, we can rewrite the above update rule of Retrace equivalently as
[TABLE]
Each here is a function of only and does not depend on , so this choice of -parameters automatically satisfies Condition 2.3(i) with the memory states being . When the discount factors are all strictly less than , for all are bounded by a deterministic constant that depends on the initial . Then for each initial , Retrace’s choice of coincides with a choice in our framework, since the -parameters in Condition 2.3(ii) can be made vacuously large so that the condition is satisfied by all the traces that could be encountered by Retrace. Thus in this case our framework for choosing effectively encompasses the particular choice used by Retrace.
One can make variations on Retrace’s trace update rule. For example, instead of truncating each importance sampling ratio by , one can truncate it by a constant , and then use a scaling scheme similar to Example 2.1 to bound the traces. The simplest such variation is to choose two memory-independent positive constants and , and replace the definition of in (2.9) by the following: with (where we treat ),
[TABLE]
Correspondingly, instead of (2.8), the update rule of becomes
[TABLE]
These variations of Retrace are similar to Example 2.1 and satisfy Condition 2.3.444To see this, let the memory states be . For each , let be defined according to (2.10), and let in Condition 2.3(ii) be (treat ). Then note that \|\lambda(y,e)e-\lambda(y,e^{\prime})e^{\prime}\|_{2}\leq\beta\min\big{\{}\tfrac{K}{\rho(s,s^{\prime})},\,1\big{\}}\cdot\|e-e^{\prime}\|_{2}\leq\beta\|e-e^{\prime}\|_{2}. ∎
2.2.2 Comparison with Previous Work
For policy evaluation, the Retrace algorithm (Munos et al., 2016) and the ABQ algorithm (Mahmood et al., 2017) are very similar (ABQ was actually developed independently of Retrace before the Munos et al. (2016) paper was published, although the ABQ paper itself was released much later). Both Retrace and ABQ include the Tree-Backup algorithm (Precup et al., 2000) as a special case. They can use additional parameters to select from a range of values, whereas Tree-Backup specifies , implicitly, in a particular way (which has the advantage of requiring no knowledge of the behavior policy) and does not have the freedom in choosing . Because of the relations between these algorithms, when comparing our method to them, we will compare it with Retrace only. In the experimental study given later in Section 4 on the performance of LSTD for various ways of setting , we will compare our scheme of choosing with that of Retrace for , which lets Retrace use the largest that it can take.
We see in Example 2.2 that the eligibility trace update rule of Retrace can be written in two equivalent forms, (2.8) and (2.9). The second form (2.9) has the advantage that the -parameters involved are shown explicitly. In TD learning, the -parameters directly affect the associated Bellman operators and can be meaningfully interpreted as stopping probabilities (see Section 3), whereas the importance sampling ratio terms in the eligibility trace iterates are essentially unchanged, for they have to be there in order to correct for the discrepancy between the behavior and target policies. For this reason, we prefer (2.9) to (2.8) and prefer thinking in terms of the selection of -parameters to that of what occurs apparently to those importance sampling ratio terms in the trace updates.
As mentioned in Example 2.2, the Munos et al. (2016) paper does not make the connection between (2.8) and (2.9). Mahmood et al. (2017) recognized the role of the -parameters and made explicit use of it to derive the ABQ algorithm. However, in the ABQ paper, the discussion and the presentation of the algorithm still emphasize the apparent changes in those importance sampling ratio terms in the trace iterates. This is an unsatisfactory point in that paper that we hope we have clarified with our present work.
We mentioned in the introduction that Retrace, ABQ and Tree-Backup are too conservative and tend to use too small values. Let us now make this statement more precise and also explain the reason behind.
These algorithms tend to behave effectively like TD() with small constant , despite that they can have at some time steps . This is due to the nature of TD learning with time-varying , which is very different from that of TD with constant . For time-varying , a large at one time step need not mean that we are using the information of the cumulative rewards over a long time horizon to estimate the value at the state encountered at time . Because the next could be very small or even zero, forcing a TD algorithm to “bootstrap” immediately. When large ’s are interleaved with small ones, we are effectively in the situation of TD with small . This could occur to our proposed scheme as well if, for example, in Example 2.1 the thresholds are set too small. When we use larger thresholds, we allow larger . By comparison, Retrace, ABQ, and Tree-Backup constrain the state-dependent -parameters to be small enough so that all the products , and this makes them prone to the small- issue just mentioned. (See the experiments in Section 4.2 for demonstrations.)
While we consider Retrace for approximate policy evaluation, the Munos et al. (2016) paper actually focuses primarily on finding an optimal policy for an MDP, in the tabular case, and it has demonstrated good empirical performance of Retrace and Tree-Backup for that purpose. Despite this, its results are not adequate yet to establish asymptotic optimality of these algorithms in the online optimistic policy iteration setting (personal communication with Munos), and it is still an open theoretical question whether online TD algorithms can solve an MDP like the Q-learning algorithm (Watkins, 1989; Tsitsiklis, 1994), when positive (small or not) and rapidly changing target policies are involved.
We also mention that for policy evaluation, Munos et al. (2016, Section 3.1) have also conceived the use of generalized Bellman operators, although they did not relate these operators explicitly to history-dependent ’s and did not study corresponding algorithms in this general case.
2.3 Ergodicity Result
The properties of the joint state-trace process are important for understanding and characterizing the behavior of our proposed TD learning scheme. We study them in this subsection. Most importantly, we shall establish the ergodicity of the state-trace process. The result will be useful in convergence analysis of several associated TD algorithms (Yu, 2017), although in this paper we discuss only the LSTD algorithm. In the next section we will also use the ergodicity result when we relate the LSTD equation (2.4) to a generalized Bellman equation for the target policy in order to interpret the LSTD solutions.
We note that to obtain the results in this subsection, we will follow similar lines of argument used in (Yu, 2012) for analyzing off-policy LSTD with constant . However, because is now history-dependent, some proof steps in (Yu, 2012) no longer apply. We shall explain this in more detail after we prove the main result of this subsection.
As another side note, one can introduce nonnegative coefficients for memory states to weight the state features (similarly to the use of “interest” weights in the ETD algorithm (Sutton et al., 2016)) and update according to
[TABLE]
The results given below apply to this update rule as well.
Let us start with two basic properties of that follow directly from our choice of the function:
- (i)
By Condition 2.3(i), for each , is a continuous function of , and thus depends continuously on . This, together with the finiteness of , ensures that is a weak Feller Markov chain.555This means that for any bounded continuous function on (endowed with the usual topology), with , \mathbb{E}\big{[}f(X_{1})\mid X_{0}=x\big{]} is a continuous function of (Meyn and Tweedie, 2009, Prop. 6.1.1).
- (ii)
Then, by a property of weak Feller Markov chains (Meyn and Tweedie, 2009, Theorem 12.1.2(ii)), the boundedness of ensured by Condition 2.3(ii) implies that has at least one invariant probability measure.
The third property, given in the lemma below, concerns the behavior of for different initial . It is an important implication of Condition 2.3(i); actually, it is our purpose of introducing the condition 2.3(i) in the first place. In the lemma, stands for “converges almost surely to.”
Lemma 2.1**.**
Let and be generated by the iteration (2.2) and (2.5), using the same trajectory of states and initial , but with different initial and , respectively. Then under Conditions 2.1(i) and 2.3(i), .
Proof.
The proof is similar to that of (Yu, 2012, Lemma 3.2). Let , and let denote the -algebra generated by . Note that under our assumption, in the generation of the two trace sequences and , the states and the memory states are the same, but the -parameters are different. Let us denote them by and for the two trace sequences, respectively. Then by (2.2), , and by Condition 2.3(i), . Hence , so \mathbb{E}\big{[}\Delta_{t}\big{|}\mathcal{F}_{t-1}\big{]}\leq\mathbb{E}\big{[}\gamma_{t}\,\rho_{t-1}\big{|}\mathcal{F}_{t-1}\big{]}\cdot\Delta_{t-1}\leq\Delta_{t-1}. This shows is a nonnegative supermartingale. By the supermartingale convergence theorem (Dudley, 2002, Theorem 10.5.7 and Lemma 4.3.3), converges a.s. to a nonnegative random variable with . From the inequality for all , we have , from which a direct calculation shows \mathbb{E}\big{[}\Delta_{t}\big{]}\leq\Delta_{0}\cdot\mathbf{1}^{\top}(P\Gamma)^{t}\mathbf{1} where denotes the -dimensional vector of all ’s. As , converges to the zero matrix under Condition 2.1(i). Therefore, and consequently, we must have a.s., i.e., . ∎
We use Lemma 2.1 and ergodicity properties of weak Feller Markov chains (Meyn, 1989) to prove the ergodicity theorem below. A direct application to LSTD will be discussed immediately after the theorem, before we give its proof.
To state the result, we need some terminology and notation. For starting from the initial condition , we write for its probability distribution, and we write “-a.s.” for “almost surely with respect to .” The occupation probability measures are denoted by , and they are random probability measures on given by
[TABLE]
where is the indicator function. We are interested in the asymptotic convergence of these occupation probability measures in the sense of weak convergence: for probability measures and on a metric space, converges weakly to if as , for every bounded continuous function .
We shall also consider the Markov chain , whose occupation probability measures are defined likewise. This Markov chain is essentially the same as , but it is more convenient for applying our ergodicity result to TD algorithms because the temporal-difference term involves . Regarding invariant probability measures of the two Markov chains, obviously, if is an invariant probability measure of , then an invariable probability measure of is the probability measure composed from the marginal and the conditional distribution of given specified by ; i.e.,
[TABLE]
(In the above, we used the notation to write the integral of w.r.t. , and the notation \zeta\big{(}d(s,y,e)\big{)} is the same as with .)
Theorem 2.1**.**
Let Conditions 2.1-2.3 hold. Then is a weak Feller Markov chain and has a unique invariant probability measure . For each initial condition of , the occupation probability measures converge weakly to , -a.s.
Likewise, the same holds for , whose unique invariant probability measure is as given in (2.13).
If the initial distribution of is , the state-trace process is stationary. Let denote expectation w.r.t. this stationary process. We now state a corollary of the above theorem for LSTD, before we prove the theorem.
Consider the sequence of equations in , , appeared in (2.4) for LSTD. From the definition (2.3) of ,
[TABLE]
we see that for fixed , every can be expressed as for a continuous function . Since the traces and hence the entire process lie in a bounded set under Condition 2.3(ii), the weak convergence of the occupation probabilities measures of shown by Theorem 2.1 implies that this sequence of equations has an asymptotic limit that can be expressed in terms of the stationary state-trace process as follows.
Corollary 2.1**.**
Let Conditions 2.1-2.3 hold. Then for each initial condition of , almost surely, the sequence of linear equations in , , tends asymptotically to (also a linear equation in ), in the sense that the random coefficients in the former equations converge to the corresponding coefficients in the latter equation as .
In the rest of this section we prove Theorem 2.1. Broadly speaking, the line of argument is as follows: We first prove the weak convergence of occupation probability measures to the same invariant probability measure, for each initial condition. This will in turn imply the uniqueness of the invariant probability measure.
After the proof we will first comment in Remark 2.1 on the differences between our proof and that of a similar result in the previous work (Yu, 2012). We will then comment in Remark 2.2 about the technical roles of Condition 2.3 (which concerns the choice of the function ) and whether some part of that condition can be relaxed.
Proof of Theorem 2.1.
As we discussed before Lemma 2.1, under Conditions 2.3, is weak Feller and has at least one invariant probability measure . Then, by (Meyn, 1989, Prop. 4.1), there exists a set with -measure such that for each initial condition , the occupation probability measures converge weakly, -a.s., to an invariant probability measure that depends only on the initial condition . To prove the theorem using this result, we need to show that (i) all these are the same invariant probability measure, and (ii) for all , has the same weak convergence property.
To this end, we first consider an arbitrary pair in the recurrent class of (cf. Condition 2.2). Let us show that for all initial conditions , converges weakly to the same invariant probability measure, almost surely.
Since the finite-state Markov chain has a single recurrent class (Condition 2.2) and its evolution is not affected by , the marginal of on coincides with the unique invariant probability distribution of . So the fact that and is a recurrent state of implies that there exists some with . For the initial condition , by the result of (Meyn, 1989) mentioned earlier, converges weakly to , almost surely.
Now consider for an arbitrary . Generate iterates and according to (2.2), using the same trajectory with , but with and . By Lemma 2.1, . Therefore, except on a null set of sample paths, it holds for all bounded Lipschitz continuous functions on that666Here we are using the same in the occupation probability measures and . This is valid because the ’s do not affect the evolution of and are functions of these states and the given initial . If we call the here instead and define using another independent copy of , then since the two sequences of occupation probability measures will have the same probability distribution, will have the same weak convergence property as .
[TABLE]
By the a.s. weak convergence of to proved earlier, except on a null set, for all such functions . Combining this with (2.14) yields that almost surely, for all such . By (Dudley, 2002, Theorem 11.3.3), this implies that almost surely, weakly.
Thus we have proved that for all initial conditions , converges weakly, almost surely, to the same invariant probability measure . Denote . Let us now show that for any initial condition , also converges to , -a.s.
Consider with an arbitrary initial condition . Let (the pair is as in the proof above). Note that a.s., because is a recurrent state of . Define , for .
By the strong Markov property (see e.g. Nummelin, 1984, Theorem 3.3), has the same probability distribution as the Markov chain that starts from . Therefore, by the preceding proof, -almost surely, for all bounded continuous functions on ,
[TABLE]
Denote . Using (2.15) and the fact a.s., we have that -almost surely,
[TABLE]
This proves that converges weakly to almost surely, for each initial condition .
It now follows that must be the unique invariant probability measure of . To see this, suppose is another invariant probability measure. For any bounded continuous function , by stationarity, for all . On the other hand, the preceding proof has established that for all initial conditions ,
[TABLE]
which implies that if is the initial distribution of , then , -a.s. We thus have
[TABLE]
where the third equality follows from the bounded convergence theorem. This shows for all bounded continuous functions , and hence by (Dudley, 2002, Prop. 11.3.2), proving the uniqueness of the invariant probability measure.
The conclusions for the Markov chain follow from the same arguments given above, if we replace with and replace the set with the set of possible state transitions. (We could have proved the assertions for first and then deduced as their implications the assertions for . We treated the latter first, as it makes the notation in the proof simpler.) ∎
Remark 2.1** (About the proof).**
Theorem 2.1 is similar to (Yu, 2012, Theorem 3.2) for off-policy LSTD with constant (the analysis given in (Yu, 2012) also applies to state-dependent ). Some of the techniques used to prove the two theorems are also similar. The main difference to (Yu, 2012) is that in the proof here we used an argument based on the strong Markov property to extend the weak convergence property of for a subset of initial conditions to all initial conditions, whereas in (Yu, 2012) this step was proved using a result on the convergence-in-mean of LSTD iterates established first. The latter approach would not work here due to the dependence of on the history. Indeed, due to this dependence, the proof of the convergence-in-mean of LSTD given in (Yu, 2012) does not carry over to our case, even though that convergence does hold as a consequence of Theorem 2.1, in view of the boundedness of traces by construction. Compared with the proof of the ergodicity result in (Yu, 2012), the proof we gave here is more direct and therefore better.
Regarding possible alternative proofs of Theorem 2.1, let us also mention that if we prove first the uniqueness of the invariant probability measure, then, since lie in a bounded set, the weak convergence of occupation probability measures will follow immediately from (Meyn, 1989, Prop. 4.2). However, because the evolution of the ’s depends on both states and traces, it does not seem easy to us to prove directly the uniqueness part first. ∎
Remark 2.2** (About the conditions on the function ).**
Our proof of Theorem 2.1 relied on Lemma 2.1 and the two properties discussed preceding that lemma, namely, that is a weak Feller Markov chain and has at least one invariant probability measure. As long as these hold when we weaken or change the conditions on the function , the proof and the conclusions of the theorem will remain applicable.
We introduced Condition 2.3(ii) to bound the traces for algorithmic concerns. For the ergodicity of the state-trace process, Condition 2.3(ii) is unimportant—in fact, it can be removed from the conditions of Theorem 2.1. The reason is that we used this condition before Lemma 2.1 to quickly infer that has at least one invariant probability measure, but this is still true without Condition 2.3(ii), in view of (Meyn and Tweedie, 2009, Theorem 12.1.2(ii)) and the fact that under Condition 2.1(i), is bounded in probability (the proof of this fact is straightforward and similar to the proof of (Yu, 2012, Lemma 3.1) or (Yu, 2015, Prop. A.1)).
Condition 2.3(i) is actually two conditions combined into one. The first is the continuity of in for each , which was used to ensure that the state-trace process is a weak Feller Markov chain. To be more general, instead of letting the evolutions of the traces and memory states be governed by the functions and , one may consider letting them be governed by stochastic kernels. Then by placing a suitable continuity condition on the stochastic kernel , one can ensure that the state-trace process has the desired weak Feller Markov property.
The second condition packed into Condition 2.3(i) is that for each , is a Lipschitz continuous function of with modulus . This condition is somewhat restrictive, and one may consider instead allowing the function to have Lipschitz modulus greater than . However, additional conditions are then needed to ensure that Lemma 2.1 holds. (If this lemma does not hold, then the state-trace process may not be ergodic and one will need a different approach than the one we took to characterize the sample path properties of the state-trace process.)
From an algorithmic perspective, if it is desirable to choose even larger ’s or to have greater flexibility in choosing these -parameters, some of the generalizations just mentioned can be considered. For example, Condition 2.3(ii) can be replaced and stochastic kernels can be introduced to allow for occasionally large traces , so that instead of having the traces bounded, one only make their variances bounded in a desired range. ∎
3 Generalized Bellman Equations
In this section, we continue the analysis started in Section 2.3. Recall that Corollary 2.1 established that the asymptotic limit of the linear equations (2.4) for LSTD is the linear equation (in ):
[TABLE]
Our goal now is to relate this equation to a generalized Bellman equation for the target policy . This will then allow us to interpret solutions of (2.4) computed by LSTD as solutions of approximate versions of that generalized Bellman equation.
To this end, we will first give a general description of randomized stopping times and associated Bellman operators (Section 3.1). We will then use these notions to derive the particular Bellman operators that correspond to our choices of the -parameters and appear in the linear equations for LSTD (Section 3.2). We will also discuss a composite scheme of choosing the -parameters as a direct application and extension of our results.
To simplify notation in subsequent derivations, we shall use the following shorthand notation: For , denote ,
[TABLE]
Also, we shall treat if .
3.1 Randomized Stopping Times and Associated Bellman Operators
Consider the Markov chain induced by the target policy . Let Condition 2.1(i) hold. Recall that for the value function , we have that for each state ,
[TABLE]
and
[TABLE]
The second equation is the standard one-step Bellman equation.
To write generalized Bellman equations for , we shall make use of randomized stopping times for , a notion that generalizes naturally stopping times for in that whether to stop at time depends not only on the past states but also on certain random outcomes. A simple example is to toss a coin at each time and stop as soon as the coin lands on heads, regardless of the history . (The corresponding Bellman equation is the one associated with TD() for a constant ; cf. Example 3.1.) Of interest here is the general case where the stopping decision does depend on the entire history.
To define a randomized stopping time formally, first, the probability space of is enlarged to take into account whatever randomization scheme that is used to make the stopping decision. (The enlargement will be problem-dependent, as the next subsection will demonstrate.) Then, on the enlarged space, a randomized stopping time for is a stopping time777A random time is called a stopping time relative to a sequence of increasing -algebras if the event for every . relative to some increasing sequence of -algebras , where the sequence is such that
- (i)
for all , (the -algebra generated by ), and
- (ii)
relative to , remains to be a Markov chain with transition probability , i.e., for all , .
See (Nummelin, 1984, Chap. 3.3); in particular, see Prop. 3.6 in p. 31-32 therein for several equivalent definitions of randomized stopping times.
Note that if for all , then the history of states fully determines whether and reduces to a stopping time for the Markov chain . The properties (i)-(ii) in the above definition encapsulate our earlier intuitive discussion about making stopping decisions, namely, stopping decisions are made based on the history and additional random outcomes that do not affect the evolution of the Markov chain.
Like stopping times, the strong Markov property also holds for randomized stopping times for a Markov chain. This is an important basic property. It says that in the event , conditioned on the -algebra associated with the stopping time relative to (which is the -algebra generated by the events that “happen before ”), the conditional distribution of is the same as the probability distribution of a Markov chain with initial state (Nummelin, 1984, Theorem 3.3).
The above abstract definition of a randomized stopping time allows us to write Bellman equations in general forms without worrying about the details of the enlarged space, which are not important at this point. For notational simplicity, when there is no confusion, we shall still write for the probability measure on the enlarged probability space and use and to denote the expectation and conditional expectation given , respectively, for that space.
If is a randomized stopping time for , the strong Markov property (Nummelin, 1984, Theorem 3.3) allows us to express in terms of and the total discounted rewards prior to stopping:
[TABLE]
where for .888In the case , . In the case , by Condition 2.1(i), is almost surely well-defined, while the second term in (3.2) is [math] because a.s., under Condition 2.1(i). Equation (3.2) is derived as follows: By the strong Markov property (Nummelin, 1984, Theorem 3.3), on ,
Then, since the term , we use the property of the conditional expectation given and the fact to rewrite this term as
\textstyle{\mathbb{E}^{\pi}_{s}\!\big{[}\,\mathbb{1}(\tau<\infty)\cdot\mathbb{E}^{\pi}\!\left[\,\sum_{t=\tau}^{\infty}\gamma_{1}^{\tau}\cdot\gamma_{\tau+1}^{t}\,r_{\pi}(S_{t})\mid\mathcal{F}_{\tau}\right]\big{]}=\mathbb{E}^{\pi}_{s}\!\left[\,\mathbb{1}(\tau<\infty)\cdot\gamma_{1}^{\tau}v_{\pi}(S_{\tau})\right]=\mathbb{E}^{\pi}_{s}\left[\gamma_{1}^{\tau}v_{\pi}(S_{\tau})\right]},
where in the last equality we also used the fact a.s. This gives (3.2).
We can also write the Bellman equation (3.2) in terms of only, by taking expectation over :
[TABLE]
where
[TABLE]
The r.h.s. of (3.2) or (3.3) defines a generalized Bellman operator associated with , which has several equivalent expressions; e.g.,
[TABLE]
Depending on the context, one expression can be more convenient to use than the other. For example, the first expression is convenient for defining through the associated and for deducing the contraction property of , whereas expressions like the second will be of interest when we want to know more explicitly the particular for our TD learning scheme and its dependence on the -parameters.
In common with one-step Bellman operator, the generalized Bellman operator is affine and involves a substochastic matrix. If a.s., then the value function is the unique fixed point of , i.e., the unique solution of , and is a sup-norm contraction. In fact, this can be shown for slightly more general :
Theorem 3.1**.**
Let Condition 2.1(i) hold, and let the randomized stopping time be such that for all states . Then is the unique fixed point of the generalized Bellman operator associated with , and is a contraction w.r.t. a weighted sup-norm on .
We prove this theorem in Appendix Appendix A: Proof of Theorem 3.1. The proof amounts to showing that if a state process evolves according to the substochastic matrix involved in the affine operator , then all the states in are transient (equivalently, the spectral radius of is less than and is invertible (Puterman, 1994, Appendix A.4)). From this the conclusions of the theorem follow as a basic fact from nonnegative matrix theory (Seneta, 2006, Theorem 1.1), and one specific choice of the weights of the sup-norm in the theorem is simply the expected time for the process to leave from each initial state (see e.g., the proof of (Bertsekas and Tsitsiklis, 1996, Prop. 2.2)).
For TD algorithms that do not use history-dependent , the random times and the corresponding Bellman operators have simple descriptions:
Example 3.1** (TD with constant or state-dependent ).**
Depending on the choice of , TD() algorithms are associated with different randomized stopping times . In the case of constant , starting from time , we stop the system with probability if it has not stopped yet; i.e.,
[TABLE]
In particular, we always stop at if , and we never stop if . Similarly, for state-dependent where , a function of the current state, the preceding stopping probability is replaced by : for . In these cases, by taking expectations over , the corresponding Bellman operators can be expressed solely in terms of and the model parameters for the target policy. ∎
3.2 Bellman Equation for the Proposed TD Learning Scheme
With the terminology of randomized stopping times, we are now ready to write down the generalized Bellman equation associated with the TD learning scheme proposed in Section 2.2. It corresponds to a particular randomized stopping time. We shall first describe this random time, from which a generalized Bellman equation follows as seen in the preceding subsection. That this is indeed the Bellman equation for our TD learning scheme will then be proved.
Consider the Markov chain under the target policy . We define a randomized stopping time for :
Let evolve according to (2.5) and (2.2):
[TABLE]
Let the initial be distributed according to , the unique invariant probability measure in Theorem 2.1 for the state-trace process induced by the behavior policy.
At time , we stop the system with probability if it has not yet been stopped. Let be the time when the system stops ( if the system never stops).
To make the dependence on the initial distribution explicit, we write for the probability measure of this process.
Note that by definition and are functions of the initial and states . From how the random time is defined, we have for all ,
[TABLE]
and hence
[TABLE]
where is the conditional distribution of given , w.r.t. the initial distribution . As before, we can write the generalized Bellman operator associated with in several equivalent forms. Let denote expectation under . Similarly to the derivation of (3.3), we can rewrite (3.2) in this case by taking expectation over conditioned on to derive that for all ,
[TABLE]
Or express in the form of (3.3) by further integrating over and using (3.7)-(3.8):
[TABLE]
for all , where in the case , and since by construction.
It will be useful later to express in terms of temporal differences. From (3.9), by writing and rearranging terms, we have for all ,
[TABLE]
In a similar way, from (3.10), we can write999Since is a randomized stopping time for the Markov chain , we have , so , i.e., . Thus we can write the term in (3.10) for as , and the expression for then follows by rearranging terms.
[TABLE]
Remark 3.1**.**
Comparing the two expressions (3.9) and (3.10) of , we remark that the expression (3.9) reflects the role of the ’s in determining the stopping time, whereas the expression (3.10), which has eliminated the auxiliary variables and , shows more clearly the dependence of the stopping time on the entire history . It can also be seen, from the initial distribution , the dependence of on the traces and the dependence of the traces on the function (which describes importance sampling ratios), that both the behavior policy and the choice of the feature representation assert a significant role in determining the Bellman operator for the target policy. This is in contrast with off-policy TD learning that uses a constant , where the behavior policy and the approximation subspace affect only how one approximates the Bellman equation underlying TD, not the Bellman equation itself, which is solely determined by (cf. Example 3.1).
Furthermore, note that as the invariant distribution of the state-trace process, is associated with the dynamic behavior of the states and traces under the behavior policy. Generally, there is no explicit expression of in terms of and the parameters in the function. As a result, in general we cannot express the operator in terms of these parameters in the learning scheme. This is different from the case of TD() where is a function of the present state only. ∎
We now proceed to show how the Bellman equation given above relates to the off-policy TD learning scheme in Section 2.2. Some notation is needed. Denote by the invariant probability measure of the Markov chain induced by the behavior policy; note that it coincides with the marginal of on . For two functions on , we write if . If is a linear subspace of functions on and for all , we write . Recall that is a function that maps each state to an -dimensional feature vector. Denote by the subspace spanned by the component functions of , which is the space of approximate value functions for our TD learning scheme. Recall also that denotes expectation w.r.t. the stationary state-trace process under the behavior policy (cf. Theorem 2.1).
Theorem 3.2**.**
Let Conditions 2.1-2.3 hold. Then as a linear equation in , \mathbb{E}_{\zeta}\big{[}e_{0}\,\delta_{0}(v)\big{]}=0 is equivalently where is the generalized Bellman operator for given in (3.9) or (3.10).
Remark 3.2** (On LSTD).**
Note that
[TABLE]
is a projected version of the generalized Bellman equation (projecting the left-hand side onto the approximation subspace w.r.t. the -weighted Euclidean norm). Theorem 3.2 and Corollary 2.1 together show that this is what LSTD solves in the limit.
Note also that although the generalized Bellman operator is a contraction (Theorem 3.1), the composition of projection with is in general not a contraction (cf. Example B.1 in Appendix Appendix B: Oblique Projection Viewpoint and Error Bound for TD). Thus we cannot use contraction-based arguments to analyze approximation properties. For that purpose, we use the oblique projection viewpoint of Scherrer (2010). Specifically, if the preceding projected Bellman equation admits a unique solution , then can be viewed as an oblique projection of (Scherrer, 2010) and the approximation error can be characterized as in (Yu and Bertsekas, 2010) by using the oblique projection viewpoint. The details of these are given in Appendix Appendix B: Oblique Projection Viewpoint and Error Bound for TD. ∎
Remark 3.3** (On gradient-based TD).**
While Theorem 3.2 is about the LSTD algorithm, it also helps in prepare the ground for analyzing gradient-based algorithms similar to those discussed in (Maei, 2011; Mahadevan et al., 2014). Like LSTD, these algorithms aim to solve the same projected generalized Bellman equation as characterized by Theorem 3.2 (cf. Remark 3.2). Their average dynamics, which is important for analyzing their convergence using the mean ODE approach from stochastic approximation theory (Kushner and Yin, 2003), can be studied based on the ergodicity result of Theorem 2.1, in essentially the same way as we did in Section 2.3 for the LSTD algorithm. For details of the convergence analysis of these gradient-based TD algorithms, see the recent work (Yu, 2017). ∎
In the rest of this subsection, we give a corollary to Theorem 3.2, deferring the proofs of both the theorem and the corollary to the next subsection. The corollary concerns a composite scheme of setting , which is slightly more general than what Section 2.2 described. It results in a Bellman operator that is a composition of the components of other Bellman operators, and it can be useful in practice for variance control. Let us describe the scheme first, before explaining our motivation for it.
Partition the state space into nonempty disjoint sets: . Associate each set with a possibly different scheme of setting that is of the type described in Section 2.2, and denote its memory states by and -function by . Keep trace vectors , one for each set, and update them according to
[TABLE]
where \lambda^{(i)}_{t}=\lambda^{(i)}\big{(}y^{(i)}_{t},e^{(i)}_{t-1}\big{)}. We then have ergodic state-trace processes that share the same state variables, \big{\{}\big{(}S_{t},y^{(i)}_{t},e^{(i)}_{t}\big{)}\big{\}}, . Each process has a unique invariant probability measure (Theorem 2.1) and an associated randomized stopping time and generalized Bellman operator , as discussed in this subsection. Define now an operator by concatenating the component mappings of for as follows: for all and ,
[TABLE]
Consider an LSTD algorithm that defines the trace to be the sum of the trace vectors,
[TABLE]
and uses the traces to form the linear equation as before,
[TABLE]
Note that is the same as By Corollary 2.1, as a linear equation in , it tends asymptotically (as ) to the linear equation \sum_{i=1}^{m}\mathbb{E}_{\zeta^{(i)}}\big{[}e_{0}^{(i)}\delta_{0}(v)\big{]}=0.
Corollary 3.1**.**
Let Condition 2.1 hold. Consider the composite scheme of setting discussed above, and let Conditions 2.2-2.3 hold for each of the schemes involved. Let LSTD calculate traces according to (3.12) and (3.14). Then the limiting linear equation (in ) associated with LSTD, \sum_{i=1}^{m}\mathbb{E}_{\zeta^{(i)}}\big{[}e_{0}^{(i)}\delta_{0}(v)\big{]}=0, is equivalently where is the generalized Bellman operator for given by (3.13) and has the same fixed point and contraction properties as stated in Theorem 3.1.
The use of composite schemes will be demonstrated by experiments in Section 4.2.2. Here let us explain informally our motivation for such schemes.
Remark 3.4** (About composite schemes of setting ).**
Our motivation for using the composite schemes is revealed by the equation (3.13). Typically each is designed to be simple to implement in TD learning. For example, if we ignore for now the bounding of traces introduced in Section 2.2 and just consider TD() with constant , can be the Bellman operator for TD() with some constant . A simple, extreme example is to partition the state space into two sets, and associate one with , and the other with . Using the combination (3.13) of the two operators in TD then means that for the first set of states whose , we want to estimate their values by using the information about the total rewards received when starting from those states, whereas for the second set of states whose , we only use the information about their one-stage rewards and how these states relate to the “neighboring” states in the transition graph. While this way of using different kinds of information for different states is natural and useful for TD-based policy evaluation, it cannot be realized by keeping a single trace sequence as before and only letting evolve with states or histories. Indeed, in that case, as discussed in Section 2.2.2, interleaving large and small ’s would make the algorithm behave effectively like TD with small over the entire state space.
In the context of the more complex scheme of setting discussed in this paper, our motivation and reasons for considering composite schemes are the same. Each can be designed to be simple to implement, such as in the simple scaling example in Section 2.2. The parameters in the th scheme can be chosen so that they encourage the use of large ’s throughout time or dictate the use of only small ’s. By combining component mappings of through (3.13), composite schemes allow us to use cumulative rewards and transition structures at different timescales for different states. This provides additional flexibility in managing the bias-variance trade-off when estimating the value function (see Figure 9 and Figure 11 in Section 4.2.2 for a demonstration).
Finally, we mention that for off-policy LSTD() with constant , composite schemes were proposed in (Yu and Bertsekas, 2012) and analyzed in (Yu, 2012, Proposition 4.5, Section 4.3). Our Corollary 3.1 extends that result. The convergence analysis of the gradient-based algorithms for the composite schemes is given in (Yu, 2017). ∎
3.3 Proofs of Theorem 3.2 and Corollary 3.1
We divide the proof of Theorem 3.2 into two steps. The first step deals with an expression for the trace vector, given in the following lemma. It is more subtle than the other step in the proof, which involves mostly calculations.
We start by extending the stationary state-trace process to , , and work with a double-ended stationary process (by Kolmogorov’s existence theorem (Dudley, 2002, Theorem 12.1.2), such a process exists). Note that as before this is a Markov chain whose transition probability is defined by the behavior policy together with the update rules (2.2) and (2.5) for , and , and the marginal distribution of each is . We keep using the notation and for this double-ended stationary Markov chain.
Recall the shorthand notation (3.1) introduced at the beginning of Section 3: For , , , , and in addition, by convention.
Lemma 3.1**.**
-almost surely, is well-defined and finite, and
[TABLE]
Proof.
First, we show \mathbb{E}_{\zeta}\big{[}\sum_{t=1}^{\infty}\gamma_{1-t}^{0}\rho_{-t}^{-1}\big{]}<\infty. Indeed,
[TABLE]
where the first equality follows from the monotone convergence theorem, the second equality from Condition 2.1(ii) and a direct calculation, and the last inequality follows from Condition 2.1(i) (since ). This implies , -a.s., so as , -a.s. Since for all , it also implies that
[TABLE]
It then follows from a theorem on integration (Rudin, 1966, Theorem 1.38, p. 28-29) that -almost surely, the infinite series converges to a finite limit.
We now prove the expression for . By unfolding the iteration (2.2) for backwards in time, we have for all ,
[TABLE]
Let in the r.h.s. of (3.16). For the last term, the trace lies in a bounded set by Condition 2.3(ii), , and as we just showed, , -a.s. So the last term converges to zero -a.s. Also as we just showed, the second term converges -almost surely to . The expression (3.15) for then follows. ∎
Proof of Theorem 3.2.
Treating , we write the expression of given in Lemma 3.1 as , -a.s. We use this expression to calculate first \mathbb{E}_{\zeta}\big{[}e_{0}\cdot\rho_{0}f(S_{0}^{1})\big{]} for an arbitrary function on . (Note that is bounded and measurable, since is finite.) We have
[TABLE]
where we used the stationarity of the double-ended state-trace process to derive the second equality, and we changed the order of expectation and summation in the first equality. This change is justified by the dominated convergence theorem, and so are similar interchanges of expectation and summation that will appear in the rest of this proof.
To proceed with the calculation, we relate the expectations in the summation in (3.17) to expectations w.r.t. the process with probability measure introduced in Section 3.2 (which we recall is induced by the target policy and involves the randomized stopping time ). Let denote expectation w.r.t. the marginal of on the space of . From the change of measure performed through , we have
[TABLE]
Combining this with (3.17) and using the fact that is the marginal distribution of in both processes, we obtain
[TABLE]
We now use (3.19) to calculate \mathbb{E}_{\zeta}\big{[}e_{0}\,\delta_{0}(v)\big{]} for a given function . Recall from (2.3) that \delta_{0}(v)=\rho_{0}\cdot\big{(}r(S_{0}^{1})+\gamma_{1}v(S_{1})-v(S_{0})\big{)}, so we let in (3.19). Since , we have
[TABLE]
where the last equality follows from the expression (3.11) for . Therefore, by (3.19),
[TABLE]
and this shows that \mathbb{E}_{\zeta}\big{[}e_{0}\,\delta_{0}(v)\big{]}=0 is equivalent to . ∎
We now prove Corollary 3.1.
Proof of Corollary 3.1.
We apply Theorem 3.2 to each state-trace process \big{\{}\big{(}S_{t},y^{(i)}_{t},e^{(i)}_{t}\big{)}\big{\}} for . Specifically, by (3.20) and the definition (3.12) of ,
[TABLE]
Hence
[TABLE]
where the last equality follows from the definition (3.13) of the operator . This shows that the linear equation in , \sum_{i=1}^{m}\mathbb{E}_{\zeta^{(i)}}\big{[}e_{0}^{(i)}\delta_{0}(v)\big{]}=0, is equivalently .
We now prove that has as its unique fixed point and is a contraction with respect to a weighted sup-norm—in other words, Theorem 3.1 applies to . For this, it suffices to show that satisfies the conditions of Theorem 3.1, namely, is a generalized Bellman operator associated with a randomized stopping time that satisfies for all states . We can define such a random time from the randomized stopping times associated with the Bellman operators . In particular, by enlarging the probability space if necessary, we can regard , , as being defined on the same probability space.101010That we can do so is clear from the definition of each as described at the beginning of Section 3.2 and from the fact that for each , the initial distribution on has the same marginal on , which is . We then let if . With this definition, we have for all states (since a.s. for all ). For each set , by (3.2), the component mappings of the generalized Bellman operator associated with are given by
[TABLE]
So by the definition (3.13) of ; i.e., is the Bellman operator associated with the randomized stopping time . ∎
4 Numerical Study
In this section, we first use a toy problem to illustrate the behavior of traces calculated by off-policy LSTD() for constant and for that evolves according to a simple special case of our proposed scheme described in Example 2.1. We then compare the behavior of LSTD for various choices of , on the toy problem and on the Mountain Car problem.
4.1 Behavior of Traces
The toy problem we use in this study has states, arranged as shown in Figure 1 (left). One state is located at the centre, and the rest of the states split evenly into four groups, indicated by the four loops in the figure. The topology of the transition graph is the same for the target and behavior policies. We have drawn the transition graph only for the northeast group in Figure 1 (left); the states in each of the other three groups are arranged in the same manner and have the same transition structure. Given this symmetry, to specify the transition matrices and for the target and behavior policies and respectively, it suffices to specify the submatrices for the central state and one of the groups. If we label the central state as state and the states in the northeast group clockwise as states -, the submatrices of and for these states are given, respectively, by
[TABLE]
Intuitively speaking, from the central state, the system enters one group of states by moving diagonally in one of the four directions with equal probability, and after spending some time in that group, eventually returns to the central state and the process repeats. The behavior policy on average spends more time wandering inside each group than the target policy, while the target policy tends to traverse clockwise through the group more quickly.
All the rewards are zero except for the middle state in each group—for the northeast group, this is the shaded state in Figure 1 (left). For the two northern (southern) groups, their middle states have reward (). The discount factor is for all states. As to features, we aggregate states into groups: the groups mentioned earlier and the central state forming its own group, and we let each state have binary features indicating its membership.
We now discuss and illustrate the behavior of traces in this toy problem. For comparison, we first do this for the off-policy TD() with a constant . It can help explain the challenges in off-policy TD learning and our motivation for proposing the new scheme of setting .
4.1.1 Traces for Constant
In this experiment we let and consider the trace iterates calculated by TD(). In general, by identifying certain cycle patterns in the transition graph, one can infer whether will be unbounded over time almost surely (Yu, 2012, Section 3.1). Figure 1 (right) shows such a cycle of states in the transition graph of the toy problem. It consists of the central state and the northeast group of states. Labeled on each edge of the cycle is the importance sampling ratio for that state transition. Traversing through the cycle once from any starting state, and multiplying together the importance sampling ratios of each edge and the discount factors of the destination states, we get . From this one can infer that calculated by off-policy TD() will be unbounded in this problem (cf. Yu, 2012, Prop. 3.1).
We plotted in the upper left graph of Figure 2 the Euclidean norm of the traces over iterations for TD(). One can see the recurring spikes and the exceptionally large values of some of these spikes in the plot. This is consistent with the unboundedness of just discussed.
The unboundedness of tells us that the invariant probability measure of the state-trace process has an unbounded support. Despite this unboundedness, is bounded in probability (Yu, 2012, Lemma 3.4) and under the invariant distribution , \mathbb{E}_{\zeta}\big{[}\|e_{0}\|\big{]}<\infty (Yu, 2012, Prop. 3.2). The latter property implies that under the invariant distribution, the probability of decreases as for large . Since the empirical distribution of the state-trace process converges to almost surely (Yu, 2012, Theorem 3.2), during a run of many iterations, we expect to see the fraction of traces with drop in a similar way as increases.
The simulation result shown in the right part of Figure 2 agrees with the preceding discussion. Plotted in the graph are fractions of traces with during iterations (the vertical axis indicates the fraction, and the horizontal axis indicates ). It can be seen that despite the recurring spikes in during the entire run, the fraction of traces with large magnitude drops sharply with the increase in .
While only a small fraction of traces have exceptionally large magnitude, they can occur in consecutive iterations. This is illustrated by the histogram in the lower left part of Figure 2. The histogram concerns the excursions of the trajectory outside of the ball . The horizontal axis indicates the lengths of the excursions (where the length is the number of iterations an excursion contains), and the vertical axis indicates how many excursions of length occurred during the iterations of the experimental run. We plotted the histogram for lengths . It can be seen that one can have large traces during many consecutive iterations. Such behavior, although tolerable by LSTD, is especially detrimental to TD algorithms and can disrupt their learning. This is our main motivation for suggesting the use of -parameters to bound the traces directly.
4.1.2 Traces with Evolving
We now proceed to illustrate the behavior of traces and LSTD for that evolves according to our proposed scheme. Specifically, for this demonstration, we will use the simple scaling example given by (2.6)-(2.7) in Example 2.1, with all the thresholds being the same constant . That is, the update rule for used in this experiment is
[TABLE]
We first simulate the state-trace process to illustrate the ergodicity of this process stated by Theorem 2.1. We will shortly study the performance of LSTD for different values of in Section 4.2.1. The results of these two experiments are shown in Figure 3 and Figures 4-5, respectively, and the details are as follows.
According to the ergodicity result of Theorem 2.1, no matter from which initial state and trace pair we generate a trajectory according to the behavior policy, the empirical distribution of state and trace pairs in this trajectory should converge, as , to the same distribution on , which is the marginal of the invariant probability measure on that space. Since is discrete, we can verify this fact by examining the empirical conditional distribution of the trace given the state. In other words, for each state , we examine the empirical distribution of the trace for the sub-trajectory , , where and it is the th visit to state by the trajectory. We check if this empirical distribution converges to the same one as we increase the length of the trajectory and as we vary the initial condition .
To give a sense of what these limiting distributions over the trace space look like, we set the parameter and generated a long trajectory with iterations. In the top row of Figure 3, we plotted three normalized histograms of the first trace component for the sub-trajectories associated with three states of the toy problem, respectively: the central state (left), the middle state of the northeast group (middle), and the first state of the southeast group (right).
To check whether the empirical conditional distributions on the trace space converge along the sub-trajectory for a given state, we compare the characteristic functions of these distributions with the characteristic function of the empirical conditional distribution obtained at the end of the sub-trajectory during the experimental run. In particular, we evaluate all these (complex-valued) characteristic functions at points, which are chosen randomly according to the multivariate normal distribution on with mean [math] and covariance matrix . We take the maximal difference between and at these points as an indicator of the deviation between the two corresponding distributions.111111Recall that a tight sequence of probability distributions on converges to a probability distribution if and only if the characteristic functions of converge pointwise to the characteristic function of (Dudley, 2002, Lemma 9.5.5). Recall also that we are dealing with convergence in distribution here, which is much weaker than convergence in total variation, so we cannot use total variation as a metric on the distribution space in this case. In the bottom row of Figure 3, the first three plots show the difference curves obtained in the way just described, for three different states, respectively. These three states are the same ones mentioned earlier in the description of the top row of Figure 3. The horizontal axis of these plots indicates , the number of visits to the corresponding state. As can be seen, the difference curves all tend to zero as increases, which is consistent with the predicted convergence of the empirical conditional distributions on the trace space.
So far we compared the empirical distributions along the same trajectory. Next we compare them against the one obtained at the end of another trajectory that starts from a different initial condition. The difference curve for one state (the first state in the southeast group) is plotted in the last graph in the bottom row of Figure 3, and it is the lower curve in that graph. As can be seen, the curve tends to zero, suggesting that the limiting distribution of these empirical distributions does not change if we vary the initial condition, which is consistent with Theorem 2.1.
For comparison, we also plotted the difference curve when these same empirical conditional distributions are compared against the empirical conditional distribution obtained from the same trajectory but for a different state (specifically, the middle state of the northeast group). This is the upper curve in the last graph in the bottom row of Figure 3. It clearly indicates that for the two states, the associated limiting conditional distributions on the trace space are different. It also shows that the characteristic function approach we adopted in this experiment can effectively distinguish between two different distributions (cf. Footnote 11).
4.2 LSTD with Evolving
We now present experiments on the LSTD algorithm.
4.2.1 A Toy Problem
Let us first continue with the toy problem of the previous subsection and show how the LSTD algorithm performs in this problem as we vary the parameter in the function for bounding the traces. We ran LSTD for , using the same trajectory, for iterations, and we computed the (Euclidean) distance of these LSTD solutions to the asymptotic TD() solution (in the space of the -parameters), normalized by the norm of the latter. We then repeat this calculation times, each time with an independently generated trajectory. Plotted in Figure 4 (left) against the values of are the means and standard deviations of the normalized distances of LSTD solutions thus obtained.
For comparison, we did the same for LSTD with a large constant : . Figure 4 (right) shows the result, where the dash-dot curve indicates the normalized distance of the asymptotic TD() solution—the solution that LSTD() would obtain in the limit. It can be seen that the performance of LSTD deteriorates as gets close to . We think that this is because due to the high variance issue, the convergence of LSTD with a large constant is too slow and requires far more iterations than the iterations performed. In comparison, LSTD with evolving behaves better: it works effectively for , and the approximation quality it achieved with such is comparable to that of asymptotic TD() solutions for a large constant around .
Figure 5 shows the quality of the asymptotic solutions of TD() with constant , for the full range of values. Plotted in the left graph is the normalized distance of the TD() solution to the TD() solution. Plotted in the right graph is the normalized approximation error of the corresponding approximate value function, where the error is measured by the weighted Euclidean norm with weights specified by (the invariant distribution on under the behavior policy), and the normalization is over the weighted norm of the true value function of the target policy. It can be seen that using provides considerably better approximations for this problem than using small .
We also found that for this problem, if we set according to the Retrace algorithm with (cf. (2.8) in Example 2.2), then the performance of LSTD is comparable to TD() with a small around . (Specifically, for Retrace, the normalized distance to the TD() solution is , and the normalized approximation error is , which are comparable to the numbers for TD(), as Figure 5 shows.) This is not surprising, because, as we discussed earlier in Section 2.2.2, in keeping always, Retrace and ABQ can be too “conservative,” resulting in an overall effect that is like using a small , even though may appear to be large at times. Recall also that this can happen to our proposed scheme too. In the present experiment, for instance, this can happen when is small; in particular, the case reduces to LSTD([math]).
4.2.2 Mountain Car Problem
In this subsection we demonstrate LSTD with evolving on a problem adapted from the well-known Mountain Car problem (Sutton and Barto, 1998). The details of this adaptation, including the target and behavior policies involved, can be found in the report (Yu, 2016a, Section 5.1, p. 23-26); most of these details are not crucial for our experiments, so to avoid distraction, we only describe briefly the experimental setup here.
In Mountain Car, the goal is to drive an underpowered car to reach the top of a steep hill, from the bottom of a valley. A state consists of the position and velocity of the car, whose values lie in the intervals , , respectively. The position corresponds to the desired hill top destination, while the position () lies at the bottom of a valley that is between the destination and a second hill peaked at in the opposite direction (see the illustration in Figure 6). Except for the destination state, each state has three available actions: , and the rewards depend only on the action taken and are , [math] and for the three actions, respectively. The dynamics is as given in (Sutton and Barto, 1998). We consider undiscounted expected total rewards, so the discount factor is except at the destination state, where the discount factor is [math] and from where the car enters a rewardless termination state permanently.
The target policy is a simple but reasonably well-behaved policy. On either slopes between the two hills, it tries to increase its energy (kinetic plus gravitational potential energy) by accelerating in the direction of its current motion. If this brings it up to the opposite hill (position ), it coasts; otherwise, if its velocity drops to near zero, it goes forward or backward with equal probability. Figure 6 visualizes the total costs of the target policy,121212The values of shown in Figure 6 are estimated by simulating the target policy for each starting state in a set of points evenly spaced in the position-velocity space. In particular, the position (velocity) interval is evenly divided into subintervals of length (), and for each stating state, the target policy is simulated times. where the horizontal (vertical) axis indicates position (velocity) and the colorbar on the right shows the value corresponding to each color. The discontinuity of the function in certain regions can be seen in this figure.
The behavior policy is an artificial policy that takes a random action (chosen with equal probability from the three actions) of the time, and explores the state space by jumping to some random state of the time. It also restarts when it is at the destination: with equal probability, it either restarts near the bottom of the valley or restarts from a random point sampled uniformly from the state space.131313The behavior policy is exactly the same as described in (Yu, 2016a, p. 24-25) except for the possibility of restarting near the bottom of the valley whenever the destination is reached.
We now explain how we will measure approximation qualities. Since the Mountain Car problem has a continuous state space, it is actually not covered by our analysis, which is for finite-state problems. Although we can treat it as essentially a finite-state problem (since the simulation is done with finite precision in computers), the number of states would still be too large to calculate the weights . So, to measure weighted approximation errors for approximate value functions produced by various LSTD algorithms in the subsequent experiments, we will compare and at a grid of points in the state space and calculate a weighted Euclidean distance between the function values at these grid points, using a set of weights precalculated by simulating the behavior policy.141414Specifically, we chose a grid of points evenly spaced in the position-velocity space. We ran the behavior policy for effective iterations, where an iteration is considered to be ineffective if the behavior policy takes an action, e.g., the restart action, that is impossible for the target policy. A visit to a state at an effective iteration was counted as a visit to the nearest grid point. At the end of the run, visits to a boundary point were disregarded as they were due to boundary effects in the dynamics of this problem, and the final counts were normalized to produce a set of weights on the grid points that sum to . The image in Figure 7 (left) visualizes these weights.
We use tile-coding (Sutton and Barto, 1998) to generate binary features for our experiments.151515Two tilings are used: the first (second) comprises of () uneven-sized rectangles that cover the state space. Together they produce a total of binary features. The details of the coding scheme are as described in (Yu, 2016a, p. 28). The approximate value functions obtained with LSTD algorithms are thus piecewise constant. For comparison, we also build a discrete approximate model by state aggregation. The discretization is done at a resolution comparable to our tile-coding scheme, and the dynamics and rewards of this model are calculated based on data collected under the behavior policy. The solution of the discrete approximate model is shown in Figure 7 (right) (the coloring scheme for this and the subsequent images are the same as shown in Figure 6). It is similar to the approximate value function calculated by LSTD([math]), which is shown in Figure 8 (first image, top row). As will be seen shortly, with positive , the approximation quality of LSTD improves. Thus the discrete model approximation approach is not as effective as the TD method in this case.
We now report the results of our experiments on the Mountain Car problem.
First Experiment: In this experiment, we compare three ways of setting : (i) Retrace with (cf. Example 2.2); (ii) our simple scaling scheme with parameter used in the previous experiments (cf. (4.1) and Example 2.1); and (iii) a composite scheme of the type discussed at the end of Section 3.2, which partitions the state space into two sets161616The first set consists of those states (position, velocity) with either or . The rest of the states belong to the second set. and applies the simple scaling scheme with parameters for the first and second set, respectively. When referring to this composite scheme in the figures, we will use the designation .
We ran LSTD with different ways of setting just mentioned, on the same state trajectory generated by the behavior policy, for effective iterations (cf. Footnote 14). Some of the approximate value functions obtained at the end of the run are visualized as images in Figure 8. It can be seen that the result of Retrace is similar to that of the simple scaling scheme with a small , and as we increase , the approximation from the scaling scheme improves.
To compare more precisely the approximation errors and see how they change over time for each algorithm, we did independent runs, each of which consists of effective iterations. The results are shown in Figures 9-11. Plotted in Figure 9 for each algorithm are the mean and standard deviation of the approximation errors for the approximate value functions obtained by that algorithm at the end of the runs. We can see from this figure the improvement in approximation quality as increases. We can also see that the result of Retrace is in between those of and , which is consistent with what the images in the top row of Figure 8 tell us.
Plotted in Figures 10-11 are the approximation errors calculated per effective iterations for each algorithm, during one of the experimental runs. Figure 10 (left) shows how Retrace compares with the simple scaling with and . It can be seen that the latter two achieved better approximation quality than Retrace without increases in variance. Figure 10 (right) shows that for larger values of , variances also became larger initially; however, after about iterations, these schemes overtook Retrace, yielding better approximations.
Figure 11 shows how the composite scheme of setting performs. Comparing the plots in this figure with the right plot in Figure 10, it can be seen that the composite schemes helped in reducing variances, and in about iterations the schemes and overtook Retrace and yielded better approximations. Together with Figure 9, Figure 11 shows clearly the bias-variance trade-off of using composite schemes in this problem.
Second Experiment: Similarly to the previous experiment, we now compare our proposed method with several other ways of setting for the LSTD algorithm as well as with a constrained variant of LSTD: (i) Retrace with as before; (ii) constant ; (iii) constrained LSTD with constant ; and (iv) the simple scaling scheme with parameter . For constant , the constrained LSTD() used in this experiment evolves the trace vectors as off-policy LSTD() does, but it forms and solves the linear equation instead, where the function truncates each component of the trace vector to be within the interval . (Such an algorithm follows naturally from the ergodicity of the state-trace process and the approximation of an unbounded integrable function by a bounded one. For a detailed discussion, see (Yu, 2016b, Section 3.2) or (Yu, 2017, Section 3.3).)
Figure 12 is similar to Figure 9 and shows, for each algorithm, the mean and standard deviation of the approximation errors of approximate value functions obtained at the end of independent experimental runs (each of which consists of effective iterations). The horizontal axis indicates the algorithms and their parameters. As can be seen from this figure, for constant small , LSTD() performed well in this problem, and LSTD() and Retrace are comparable. For constant , LSTD() failed to give sensible results, and LSTD() started to show this unreliable behavior. This behavior of LSTD() is related to what we observed in Figure 4(right) in the small toy problem, and it is, we think, due to the high variance issue, which becomes more severe as gets larger. Constrained LSTD() is much more reliable, and it did consistently well for all values of tested. LSTD with evolving also did well, and with , it achieved a slightly better approximation quality than constrained LSTD().
Figure 13 compares the temporal behavior of several algorithms during one experimental run. Plotted are the approximation errors calculated per iterations for each algorithm in the comparison. It can be seen that in this problem constrained LSTD() did not suffer from large variances even with large values, and compared with constrained LSTD(), the behavior of LSTD with evolving was also reasonable for large .
Third Experiment: In this experiment we first compare the simple scaling scheme and Retrace, where both schemes now use an additional parameter , as discussed in Examples 2.1-2.2. Recall that when , they reduce to the schemes that we already compared in the previous experiments. For both schemes, as becomes smaller, we expect the approximation quality to drop but the variance to get smaller.
Plotted in Figure 14 for and Retrace are the results obtained from a single experimental run consisting of effective iterations. As before the approximation errors were calculated per iterations. The results do show the expected bias-variance trade-off effects of the parameter , during the initial period of the experimental run, although the effects on Retrace turned out to be smaller and hard to discern at the scale of the plot.
Next we test some of the variations on Retrace discussed in Example 2.2. Specifically, we consider (2.10)-(2.11) with parameters , and . Plotted in Figure 15 are the results from one experimental run of effective iterations. For comparison, the plots also show the behavior of Retrace and the simple scaling scheme with the same values of during that run (these algorithms used the same ). As expected and can be seen from the figure, the approximation quality improves with and . While the variances also tend to increase during the initial part of the run, the variants that truncate the importance sampling ratios by performed comparably to Retrace initially, soon overtook Retrace and achieved better approximation quality.
5 Conclusion
We developed in this paper a new scheme of setting the -parameters for off-policy TD learning, using the ideas of randomized stopping times and generalized Bellman equations for MDPs. Like the two recently proposed algorithms Retrace (Munos et al., 2016) and ABQ (Mahmood et al., 2017), our scheme keeps the traces bounded to reduce variances, but it is much more general and flexible. To study its theoretical properties, we analyzed the resulting state-trace process and established convergence and solution properties for the associated LSTD algorithm, and these results have prepared the ground for convergence analysis of the gradient-based implementation of our proposed scheme (Yu, 2017). In addition we did a preliminary numerical study. It showed that with the proposed scheme LSTD can outperform several existing off-policy LSTD algorithms. It also demonstrated that in order to achieve better bias-variance trade-offs in off-policy learning, it is helpful to have more flexibility in choosing the -parameters and to allow for large values. Future research is to conduct a more extensive numerical study of both least-squares based and gradient-based algorithms, with more versatile ways of using the memory states and -parameters, in off-policy learning applications.
Acknowledgements
An abridged version of this article appeared first at The th Canadian Conference on Artificial Intelligence (CAI), May 2017, and a reformatted copy is to appear in Journal on Machine Learning Research (JMLR), 2018. We thank several anonymous reviewers for CAI and JMLR for their helpful feedback. We also thank Dr. Ajin Joseph and Dr. Martha Steenstrup for reading parts of our manuscript and providing comments that helped improve our presentation. This research was supported by a grant from Alberta Innovates—Technology Futures.
\appendixpage
Appendix A: Proof of Theorem 3.1
We prove Theorem 3.1 in this appendix. Recall from Section 3.1 that the generalized Bellman operator associated with a randomized stopping time satisfies that . By (3.2), the substochastic matrix in this affine operator is given by
[TABLE]
We can extend to a transition matrix by adding an additional absorbing state to the system, so that and
[TABLE]
Both conclusions of Theorem 3.1 will follow immediately if we show that is invertible. Indeed, if is invertible, then has a unique solution, which must be since is always a solution of this equation. In addition, if is invertible, then since is a substochastic matrix, the spectral radius of must be less than . Consider adding to a small enough perturbation , where is the matrix of all ones and is a sufficiently small positive number so that the spectral radius of is less than . Applying (Seneta, 2006, Theorem 1.1) to the nonnegative primitive matrix , we have that where is a positive eigenvector of the matrix corresponding to a positive eigenvalue that is strictly less than . Consequently , implying that is a linear contraction w.r.t. a weighted sup-norm (with weights ), which is the second conclusion of the theorem.
Hence, to prove Theorem 3.1, it suffices to show that the inverse exists, which is equivalent to that for the Markov chain on with transition matrix , all the states in are transient (see e.g., Puterman 1994, Appendix A.4).
We prove this by contradiction. Suppose it is not true, and let be a recurrent class of the Markov chain. Then for all , for (i.e., the submatrix of corresponding to is a transition matrix). In particular, , so
[TABLE]
implying that given , a.s. (since ). Then by (A.1),
[TABLE]
Observe from (A.2) that given , the event has zero probability. This is because under Condition 2.1(i), (as ) and consequently a.s. Indeed, the existence of the inverse under Condition 2.1(i) implies as . Since the th entry of equals , we have as . As the nonnegative sequence is nonincreasing (since each ), this implies, by Fatou’s lemma (Dudley, 2002, Lemma 4.3.3), that . Thus if , is almost surely finite.
We now consider a Markov chain with transition matrix and . We will extract from it a Markov chain with transition matrix on the recurrent class , by employing multiple stopping times. We will show a.s., contrary to the fact a.s. just discussed.
For ease of explanation, let us imagine that there is a device that if we give it a sequence of states generated according to , it will output the stopping decision at a random time that is exactly the randomized stopping time associated with the operator . (Because is a randomized stopping time for , a device that correctly implements does not affect the evolution of , so we can give to the device as inputs.)
Let start from some state . We use the device just mentioned to generate the first randomized stopping time . We then reset the device so that it now “sees” the time-shifted process with the initial state being . We wait till the device makes another stopping decision, and we designate that time by . We repeat this procedure as soon as the device makes yet another stopping decision. This gives us a nondecreasing sequence .
Since , by what we proved earlier, is almost surely finite. Let and . The transition from to is according to the transition matrix by construction (cf. (A.3)). Since is a recurrent class for , we must have almost surely. Then, repeating the same argument and using induction, we have that almost surely, for all , is defined and finite and . Thus we obtain an infinite sequence , which is a recurrent Markov chain on with its transition matrix given by the corresponding submatrix of .
Now in view of (A.2), almost surely,
[TABLE]
(recall that if , by definition). Consider first a simpler case of the randomized stopping time that defines : for any initial state , a.s. Then, is strictly increasing, and by multiplying the variables in (A.4) together, we have a.s. For the general case of assumed in the theorem, is possible, but for all states . This means that the event of being the same for all greater than some (random) has probability zero. So must converge to almost surely, and we again obtain, by multiplying the variables in (A.4) together, that a.s. This contradicts the fact proved earlier, namely, that for any initial state , a.s. So the assumption of a recurrent class for must be false. This proves Theorem 3.1.
Appendix B: Oblique Projection Viewpoint and Error Bound for TD
In this appendix we first explain Scherrer’s interpretation of TD solutions as oblique projections (Scherrer, 2010), and we then give approximation error bounds for TD similar to those given by Yu and Bertsekas (2010), which do not rely on contraction properties. We will explain these properties of TD in the context of generalized Bellman operators discussed in this paper. Although this was not the framework used in (Scherrer, 2010; Yu and Bertsekas, 2010) and our setup here is more general than the one discussed in those previous papers, the arguments and reasoning are essentially the same.
B.1 Solutions of TD as Oblique Projections of the Value Function
Let us start with the projected Bellman equation associated with TD/LSTD that we noted in Remark 3.2:
[TABLE]
where is a generalized Bellman operator with as its unique fixed point, and is the approximation subspace. We can write this equation equivalently as
[TABLE]
where denotes the projection onto the approximation subspace with respect to the -weighted Euclidean norm. In the subsequent derivations, we will not use the fact that is the invariant probability measure induced by the behavior policy on , so the analyses we give in this appendix apply to any weighted Euclidean norm .
Scherrer (2010) first realized that the solution of the projected Bellman equation (B.1) can be viewed as an oblique projection of the value function on the approximation subspace . This viewpoint provides an intuitive geometric interpretation of the TD solution and explains conceptually the source of its approximation bias. Analytically, this view also gives tight bounds on the approximation bias, as we will elaborate later in Section B.2.
An oblique projection is defined by two nonorthogonal subspaces of equal dimensions: it is the projection onto the first subspace orthogonally to the second (Saad, 2003), as illustrated in Figure 16. More precisely, for any two -dimensional subspaces of such that no vector in is orthogonal to , there is an associated oblique projection operator defined by
[TABLE]
If the two subspaces are the same: or if lies in , then the oblique projection is the same as , the orthogonal projection of onto . In general this need not be the case and typically (cf. Figure 16). A matrix representation of the projection operator is given by
[TABLE]
where and are matrices whose columns form a basis of and , respectively (see Saad 2003, Chap. 1.12). For comparison, a matrix representation of the orthogonal projection operator is \Pi_{\mathcal{L}_{1}}=\Phi_{1}\big{(}\Phi_{1}^{\top}\Phi_{1}\big{)}^{-1}\Phi_{1}^{\top}. (Below we will use the same notation for a projection operator and its matrix representations.)
Back to the projected Bellman equation (B.1), let us assume it has a unique solution and express in terms of . We have . By Theorem 3.1, is the unique solution of . Recall that the generalized Bellman operator is affine and can be expressed as for some vector and substochastic matrix . It follows that and
[TABLE]
Let the columns of form a basis of the approximation subspace , and let be the diagonal matrix with as its diagonal elements. Then a matrix representation of the projection operator is \Pi_{\zeta_{\mathcal{S}}}=\Phi\big{(}\Phi^{\top}D\,\Phi\big{)}^{-1}\Phi^{\top}D. Using this representation of and the fact , we obtain from (B.4) an expression of in terms of :
[TABLE]
(Here the invertibility of the matrix is equivalent to our assumption that is the unique solution of (B.1).)
Let us compare the expression (B.5) with (B.3). We see that if the geometry on is determined by the usual Euclidean norm , then is an oblique projection of for the two subspaces and \mathcal{L}_{2}=\text{column-space}\big{(}(I-\tilde{P}_{\pi})^{\top}D\Phi\big{)}.
Alternatively, consider the case where the geometry on is determined by some weighted Euclidean norm with weights (for example, , which is one of the cases of interest in off-policy learning). In this case we can first scale each coordinate by the square root of its weight to reduce the case to that of the norm . Specifically, let denote orthogonality with respect to , and let denote the diagonal matrix that has as its diagonal elements. For the linear mapping , we have
[TABLE]
The second relation means that is an oblique projection of for two subspaces with respect to (i.e., and ), if and only if is an oblique projection of for the two subspaces with respect to :
[TABLE]
With these facts in mind, we rewrite (B.5) equivalently as follows:
[TABLE]
where
[TABLE]
and applied to a matrix denotes the result of applying to each column of that matrix. Comparing (B.7) with (B.3), we see that is an oblique projection of with respect to for the two subspaces , , where
[TABLE]
So based on the discussion earlier, with respect to the weighted Euclidean norm on , is an oblique projection of for the two subspaces .
Note that by (B.8), the second subspace defining the above oblique projection is the image of the approximation subspace under the linear transformation . Thus depends on the dynamics induced by the target policy as well as the generalized Bellman operator that we choose. Relating the oblique projection interpretation of to Figure 16, we can see where the approximation bias of TD, , comes from.
B.2 Approximation Error Bound
We now consider the approximation error of and use the oblique projection viewpoint to derive a sharp bound on the approximation bias . Before proceeding, however, let us first remind the reader that unless the norm for the projection operator is purposefully chosen, the composition of with a generalized Bellman operator is usually not a contraction, and thus error bounds for projected generalized Bellman equations usually cannot be obtained with contraction-based arguments. This is the case even for on-policy learning, as the following example shows.
Example B.1** (Non-contractive ).**
If the target policy induces an irreducible Markov chain with invariant probability measure , and if is the Bellman operator for TD() with a constant , then, as Tsitsiklis and Van Roy (1997) showed, is a contraction operator w.r.t. the weighted Euclidean norm . Consequently, the matrix associated with the TD() algorithm is negative definite (Tsitsiklis and Van Roy, 1997). The derivation of the contraction property of in this case relies critically on the inequality . This inequality generally does not hold for the substochastic matrix in the generalized Bellman operator , when is not constant. So, for non-constant , we can no longer expect to be a contraction or the matrix to be negative definite.
As an example, consider a simple two-state problem in which the system under the target policy moves from one state to another in a cycle. Let , let the discount factor be a constant, and let be a function of states with , . Then and . For near , e.g., , and for as given below, we can calculate the -weighted norm of and the matrix associated with TD():
[TABLE]
The latter matrix is not negative definite; in fact, its eigenvalues have positive real parts, so it is not even a Hurwitz matrix (TD() can diverge in this case). In the above, we have , and while is not a contraction w.r.t. , it is still a contraction w.r.t. some matrix norm. If we now let instead, then the spectral radius of the matrix comes out as , so (and hence ) cannot be a contraction w.r.t. any matrix norm. ∎
We now proceed to bound the bias term relative to , the distance between and the approximation subspace measured with respect to . It is more transparent to derive the bound for the case of a general oblique projection operator with respect to the usual Euclidean norm , so let us do that first and then use the linear transformation to translate the result to TD, as we did earlier in the preceding subsection.
We bound the bias relative to , by calculating
[TABLE]
(where we treat ). This constant depends on the two subspaces , , and reflects the “angle” between them. It has several equivalent expressions, e.g.,
[TABLE]
or with denoting the spectral radius of a square matrix ,
[TABLE]
(After the definition of , each expression of in (B.10)-(B.12) follows from the preceding one; in particular, for the last expression in (B.12), we used the fact that , and .)
We can express in terms of the spectral radius of an matrix, similarly to what was done in (Yu and Bertsekas, 2010). In particular, we take the last expression of in (B.12) and rewrite the symmetric matrix in that expression using the matrix representations of the projection operators as follows:
[TABLE]
By a result in matrix theory (Horn and Johnson, 1985, Theorem 1.3.20), for any matrix and matrix , . Applying this result to the preceding expression with , we have that \sigma\big{(}\Pi_{\mathcal{L}_{1}\mathcal{L}_{2}}\Pi_{\mathcal{L}_{2}\mathcal{L}_{1}}-\Pi_{\mathcal{L}_{1}}\big{)} is equal to the spectral radius of the matrix
[TABLE]
Combing this with (B.12), we obtain that
[TABLE]
Thus, for the above , the bound below holds for all and with equality attained at some :
[TABLE]
We now translate the result (B.13)-(B.14) to our TD context. We want to bound the relative bias \big{\|}v_{\text{\tiny TD}}-\Pi_{\xi}v_{\pi}\big{\|}_{\xi}/\big{\|}v_{\pi}-\Pi_{\xi}v_{\pi}\big{\|}_{\xi}. As discussed earlier in Section B.1, we can replace with by using the linear transformation to scale the coordinates. In particular, by the two relations given in (B.6),
[TABLE]
where on the r.h.s. stands for the orthogonal projection onto with respect to . As shown by (B.7), is an oblique projection of for the two subspaces and , where is given by (B.8)-(B.9). According to (B.13), the constant for this oblique projection is where, if we take and as defined by (B.8), is now given by the expression in (B.13) with in place of , respectively. Thus, by (B.14) and (B.15) we obtain that
[TABLE]
where is an matrix given by
[TABLE]
for
[TABLE]
or more explicitly, after substituting the expressions of and in the formula of and removing , we have
[TABLE]
Note that by the definition of , the bound (B.16) is a worst-case bound that depends only on the two subspaces involved in the oblique projection operator. In other words, given the approximation subspace , the dynamics described by , the projection norm and the norm for measuring the approximation quality, the bound (B.16) is attained by a worst-case choice of the rewards for the target policy. In this sense, the bound (B.16) is tight.
The reference list from the paper itself. Each links out to its DOI / PubMed record.
- 1Bertsekas and Tsitsiklis (1996) Bertsekas, D. P. and Tsitsiklis, J. N. (1996). Neuro-Dynamic Programming . Athena Scientific, Belmont, MA.
- 2Borkar (2008) Borkar, V. S. (2008). Stochastic Approximation: A Dynamic Viewpoint . Cambridge University Press, Cambridge.
- 3Boyan (1999) Boyan, J. A. (1999). Least-squares temporal difference learning. In The 16th Int. Conf. Machine Learning (ICML) .
- 4Dai et al. (2018) Dai, B., Shaw, A., Li, L., Xiao, L., He, N., Liu, Z., Chen, J., and Song, L. (2018). SBEED: Convergent reinforcement learning with nonlinear function approximation. In The 35th Int. Conf. Machine Learning (ICML) .
- 5Dann et al. (2014) Dann, C., Neumann, G., and Peters, J. (2014). Policy evaluation with temporal differences: A survey and comparison. Journal of Machine Learning Res. , 15:809–883.
- 6Dudley (2002) Dudley, R. M. (2002). Real Analysis and Probability . Cambridge University Press, Cambridge.
- 7Geist and Scherrer (2014) Geist, M. and Scherrer, B. (2014). Off-policy learning with eligibility traces: A survey. Journal of Machine Learning Res. , 15:289–333.
- 8Glynn and Iglehart (1989) Glynn, P. W. and Iglehart, D. L. (1989). Importance sampling for stochastic simulations. Management Science , 35:1367–1392.