Conditions on Features for Temporal Difference-Like Methods to Converge

TL;DR

This paper characterizes when reinforcement learning algorithms with linear function approximation converge or not, revealing that feature choices critically influence convergence and proposing conditions for safe feature selection.

Contribution

It introduces a new feature condition determining convergence of natural RL algorithms, unifies counter-examples, and guides feature design for reliable convergence.

Findings

Natural algorithms converge if all value functions satisfy a specific shape.

Most feature choices can lead to convergence to incorrect solutions.

State aggregation features are proven to be a safe choice for convergence.

Abstract

The convergence of many reinforcement learning (RL) algorithms with linear function approximation has been investigated extensively but most proofs assume that these methods converge to a unique solution. In this paper, we provide a complete characterization of non-uniqueness issues for a large class of reinforcement learning algorithms, simultaneously unifying many counter-examples to convergence in a theoretical framework. We achieve this by proving a new condition on features that can determine whether the convergence assumptions are valid or non-uniqueness holds. We consider a general class of RL methods, which we call natural algorithms, whose solutions are characterized as the fixed point of a projected Bellman equation (when it exists); notably, bootstrapped temporal difference-based methods such as $TD(\lambda)$ and $GTD(\lambda)$ are natural algorithms. Our main result proves that natural algorithms converge to the correct solution if and only if all the value functions in the approximation space satisfy a certain shape. This implies that natural algorithms are, in general, inherently prone to converge to the wrong solution for most feature choices even if the value function can be represented exactly. Given our results, we show that state aggregation based features are a safe choice for natural algorithms and we also provide a condition for finding convergent algorithms under other feature constructions.

Equations

$$V(s)=R(s)+\gamma {\int }_{\mathcal{S}}T(s^{\prime }|s)V(s)d{s}^{\prime },$$

$$\left(\mathcal{T}V\right)(s)=R(s)+\gamma {\int }_{\mathcal{S}}T(s^{\prime }|s)V(s^{\prime })d{s}^{\prime }.$$

$$⟨f,g{⟩}_{\mu }:={\int }_{\mathcal{S}}f(s)g(s)\mu (s)ds.$$

$$({\mathcal{T}}^{(\lambda )}V)(s):=$$

$$\displaystyle\cdot\mathbb{E}\left[\sum_{t=0}^{m}\gamma^{t}R(s_{t})+\gamma^{m+1}V(s_{t+1})\bigg{|}s_{0}=s\right]$$

$${\mathcal{T}}^{(\lambda )}V=(1-\lambda )\sum _{m=0}^{\infty }{\lambda }^m\left(\sum _{t=0}^m(\gamma P_T{)}^{t}R+(\gamma P_T{)}^{m+1}V\right)$$

$${\mathcal{T}}^{(\lambda )}V=R^{(\lambda )}+\gamma P_T^{(\lambda )}V,$$

$$R^{(\lambda )}:=(1-\lambda )\sum _{m=0}^{\infty }{\lambda }^m\sum _{t=0}^m(\gamma P_T{)}^{t}R,$$

$$P_T^{(\lambda )}:=(1-\lambda )\sum _{m=0}^{\infty }(\lambda \gamma {)}^m(P_T{)}^{m+1}.$$

$$G:=\frac{(1-\lambda )\gamma }{1-\lambda \gamma }.$$

$$\hat{V}=\Pi {\mathcal{T}}^{(\lambda )}\hat{V}$$

$$\hat{V}=\Pi {\mathcal{T}}^{(\lambda )}\hat{V}.$$

$$A_{ij}:=⟨{\psi }_i,{\varphi }_j{⟩}_{\mu },B_{ij}:=⟨{\psi }_i,P_T^{(\lambda )}{\varphi }_j{⟩}_{\mu },b_i:=⟨{\psi }_i,R{⟩}_{\mu }$$

$$\left(A-\gamma B\right)w=b.$$

$$\hat{V}=\Pi {\mathcal{T}}^{(\lambda )}\hat{V},$$

$$T_1=\left[\begin{array}{cc}0& 1\\ \frac{1}{2}& \frac{1}{2}\end{array}\right],T_2=\left[\begin{array}{ccc}0& \frac{1}{2}& \frac{1}{2}\\ 1& 0& 0\\ 0& \frac{1}{2}& \frac{1}{2}\end{array}\right].$$

$$⟨{\psi }_i,\phi {⟩}_{\mu }\ge G{\phi }_{\max }\text{ or }⟨{\psi }_i,\phi {⟩}_{\mu }\le G{\phi }_{\min }.$$

$$\hat{V}=\Phi {\Phi }^{†}\mathcal{T}(\hat{V}),$$

$$\hat{V}=R+\gamma P_T^{(\lambda )}\hat{V},$$

$$⟨{\psi }_i,{\varphi }_j{⟩}_{\mu }=A_{ij},⟨{\psi }_i,P_T^{(\lambda )}{\varphi }_j{⟩}_{\mu }=B_{ij},⟨{\psi }_i,R{⟩}_{\mu }=b_{ij}$$

$$(A-\gamma B)w=b.$$

$${\int }_{s\in \mathcal{S}}{\chi }_i(s)\phi (s)ds=G{\int }_{s\in \mathcal{S}}{\chi }_i(s)f(s)ds.$$

$$\bar{f}(s):=\frac{1}{G}\phi (s)-\delta (s),$$

$${\int }_{\mathcal{S}}{\chi }_i(s)\delta (s)ds={\int }_{\mathcal{S}}{\chi }_i(s)g(s)ds$$

$$(A-\gamma B)w=b.$$

$$w^{\xi }=w^{\ast }+\xi v,T^{\xi }=T^{\ast },R^{\xi }={\varphi }^{\top }{w}^{\xi }-\gamma P_{T^{\xi }}^{(\lambda )}{\varphi }^{\top }{w}^{\xi }.$$

$${\mathcal{T}}^{(\lambda )}{\varphi }^{\top }(s)w$$

$$={\varphi }^{\top }(s)w^{\xi },$$

$$A_{ij}^{\xi }=⟨{\psi }_i,{\varphi }_j{⟩}_{\mu },B_{ij}^{\xi }=⟨{\psi }_i,P_{T^{\xi }}^{(\lambda )}{\varphi }_j{⟩}_{\mu },b_i^{\xi }=⟨{\psi }_i,R{⟩}_{\mu },$$

$$\left(A^{\xi }-\gamma B^{\xi }\right)w^{\xi }=b^{\xi }.$$

$$b^{\xi }$$

$$=\left(A^{\xi }-\gamma B^{\xi }\right)\left(w^{\ast }+\xi v\right)$$

Full text

1 Introduction

A longstanding goal in reinforcement learning (RL) has been to find algorithms with linear function approximation that reliably converge to the fixed point of the Bellman equations. As such the convergence of different RL methods that display such characteristics have been researched extensively. The $TD(\lambda)$ algorithm converges in an on-policy learning setting to the fixed point of a projected $\lambda$-weighted Bellman equation TVR (97). The Residual Gradient algorithm is shown to minimise the Bellman error but suffers from double sampling Bai (95). More recently, some temporal difference-based methods have been shown to converge with off-policy learning. The GTD2 and TDC algorithms are shown to converge to the $TD(0)$ solution under an off-policy learning setting SMP*+* (09). These algorithms have also been extended to their bootstrapped version $GTD(\lambda)$ and shown to converge to the $TD(\lambda)$ solution Mae (11). However, a core tenet in almost all of these convergence results is the assumption that these RL methods converge to a unique solution; for example in the proof of GTD2’s convergence, the matrix quantities $A$ and $C$ are assumed to be non-singular, allowing for uniqueness of solution (SMP*+*, 09, Theorem 1).

In addition to the convergence results, pertinent counter-examples have been documented in the literature that highlight how the choice of features is crucial to convergence of RL methods Gor (95); TVR (96); Bai (95); Ber (95); BM (95). Bertsekas showed that $TD(\lambda)$ with function approximation may converge to a parameter vector which generates a poor estimate of the value function (in terms of Euclidean distance) Ber (95). Tsitsiklis and Van Roy provided a counter-example showing that RL methods may diverge even when the value function is representable by the chosen features TVR (96). More recently, Sutton and Barto present a counter-example where methods that minimise the Bellman error may fail to learn the correct parameter value (SB, 18, Example 11.4).

In this paper, we provide a complete characterization of non-uniqueness and the potential to converge to the wrong solution for a large class of RL algorithms we call natural algorithms. A natural algorithm is any method that can be characterized as solving for the unique fixed point (when it exists) of a projected Bellman equation. We consider all oblique projections and a Bellman equation based on the $TD(\lambda)$ Bellman operator presented in TVR (97). Under this definition, the natural algorithms include a large spectrum of algorithms: on one end of the spectrum, the natural algorithms include bootstrapped methods such as $TD(\lambda)$ and $GTD(\lambda)$ since they are characterized by an orthogonal projection, and on the other end the natural algorithms include Bellman-error based methods which are characterized by the identity projection. We consider an RL setting with a continuous state space $\mathcal{S}$ and finite action space $\mathcal{A}$; note that a finite state space is a special case of our setup. Furthermore, we consider the infinite horizon problem for our results.

Our main contribution is to prove that natural algorithms, even under the setting where the value function can be represented exactly by the features, are inherently prone to non-uniqueness and will converge to the wrong solution for most feature choices. Our main result is as follows:

Theorem 5.1.

Natural algorithms converge if and only if all non-zero linear combination of the features achieve their extreme values on a sub-region of the state space that has non-zero measure under the stationary distribution.

Importantly, given our characterisation, we provide some guidelines for choosing features and algorithms to avoid non-uniqueness. We show that state aggregation based features are a safe choice. We also provide a sufficient condition for algorithms to converge under other feature constructions.

This paper is organized as follows. In Section 2, we introduce some background and notation. In Section 3, we present the theory behind projected equation methods and the characteristic equation to projected Bellman equations. In Section 4, we present a detailed look at the counter-example presented by Sutton and Barto that demonstrates the non-uniqueness issues which plague Bellman-error methods SB (18). In Section 5, we present our main results and discuss their implications, including positive feature construction examples. In Section 6, we present our framework for analyzing convergence. Finally, in Section 7, we present the idea behind the proof of Theorem 5.1. For brevity, most proofs and supporting results have been omitted. However, the supporting results and omitted proofs can be found in Appendices A and B respectively.

2 Background and Notation

We now reiterate some background RL concepts, mathematical concepts and notation used throughout this paper.

2.1 RL in Continuous State Space

We consider an agent-environment setup SB (18) where an agent follows a stationary policy $\pi$ and interacts with a Markov Decision Process (MDP). We assume a continuous state space $\mathcal{S}$ that is compact and measurable and a finite action space $\mathcal{A}$. For simplicity, we will assume that $\mathcal{S}=\mathbb{R}$ in all our examples. The expected reward function is a function $R\mathrel{\mathop{\mathchar 58\relax}}\mathcal{S}\to\mathbb{R}$ and represents the expected reward to be received for a given state following $\pi$. At a state $s\in\mathcal{S}$, we assume that there is a transition density function $T\mathrel{\mathop{\mathchar 58\relax}}\mathcal{S}\times\mathcal{S}\to[0,1]$ whilst following $\pi$. Combined with an initial state $s_{0}$, the state sequence can be viewed as a time-homogeneous Markov process with transition kernel defined by $T(B|x)=\int_{B}T(y|x)dy~{},\forall B\in\mathcal{B}(\mathcal{S})~{},x\in\mathcal{S}$ where $\mathcal{B}(\mathcal{S})$ is the Borel sigma-algebra. We consider the infinite horizon problem and thus the value function at state $s\in\mathcal{S}$ is defined as the total discounted expected return: V(s)=\mathbb{E}\left[\sum_{t=0}^{\infty}\gamma^{t}R(s_{t})\bigg{|}s_{0}=s\right], where $\gamma\in[0,1)$ is the discount factor. By standard MDP theory, the value function satisfies the Bellman equation given by

[TABLE]

for any state $s\in\mathcal{S}$. The Bellman operator $\mathcal{T}\mathrel{\mathop{\mathchar 58\relax}}\mathbb{R}^{\mathcal{S}}\to\mathbb{R}^{\mathcal{S}}$ is an affine linear operator on $\mathbb{R}^{S}$ and is defined accordingly as

[TABLE]

If we define $P_{T}$ to be an operator such that $(P_{T}f)(s)\coloneqq\int_{\mathcal{S}}T(s^{\prime}|s)f(s^{\prime})ds^{\prime}$, we can express the Bellman operator compactly as $\mathcal{T}V\coloneqq R+\gamma P_{T}V$ for any $V\in\mathbb{R}^{\mathcal{S}}$. The Bellman equation can then be expressed as the fixed point equation $V=\mathcal{T}V$.

For an agent following a policy $\pi$ and interacting with an MDP, the state sequence can be viewed as a Markov process with transition density function $T$. Throughout this paper we assume that the state Markov process admits a stationary measure $\mu$. Under these assumptions, the value function space inherits extra geometric structure via an inner product defined with respect to $\mu$. For any $f,g\in\mathbb{R}^{\mathcal{S}}$,

[TABLE]

Showing that $\langle\cdot,\cdot\rangle_{\mu}$ is an inner product is routine. We define the norm on the associated inner product space by $\mathinner{\!\left\lVert\cdot\right\rVert}_{\mu}=\sqrt{\langle\cdot,\cdot\rangle_{\mu}}$. The set of functions in the value function space with finite $\mathinner{\!\left\lVert\cdot\right\rVert}_{\mu}$-norm is given by $L^{2}(\mathcal{S},\mu)\coloneqq\{V\in\mathbb{R}^{\mathcal{S}}\mathrel{\mathop{\mathchar 58\relax}}\mathinner{\!\left\lVert V\right\rVert}_{\mu}<\infty\}$. Under our assumptions, it can be shown that the value function associated with the Markov process lives in $L^{2}(\mathcal{S},\mu)$. Furthermore, our approximations $\hat{V}$ of $V$ also evolve in this space. For any two functions $V,\tilde{V}\in L^{2}(\mathcal{S},\mu)$ we say that $V$ is $\mu$-orthogonal to $\tilde{V}$, denoted by $V\perp_{\mu}\tilde{V}$, if and only if $\langle V,\tilde{V}\rangle_{\mu}=0$.

2.2 Linear Function Approximation

When using linear function approximation, the value function $V$ is approximated by a linear combination of the features chosen from a finite-dimensional subspace of $L^{2}(\mathcal{S},\mu)$. Formally, the approximate value function $\hat{V}$ can be written as a linear combination $\hat{V}(s,w)=\sum_{i=0}^{k}\phi_{i}(s)w_{i},~{}\forall s\in\mathcal{S}$ where $w=(w_{1},w_{2},\ldots,w_{k})^{\top}\in\mathbb{R}^{k}$ is a parameter vector and $\Phi=\left\{\phi_{1},\ldots,\phi_{k}\right\}$ is the set of features from $\mathcal{S}$ to $\mathbb{R}$ such that span$(\Phi)$ is a finite-dimensional subspace of $L^{2}(\mathcal{S},\mu)$. To simplify notation, let us define $\phi(s)=(\phi_{1}(s),\phi_{2}(s),\ldots,\phi_{k}(s))^{\top}$. The approximation $\hat{V}$ can then be represented compactly as an Euclidean inner product $\hat{V}(s,w)=\langle\phi(s),w\rangle=\phi^{\top}(s)w$.

We now define a property of linear combination of features that will be crucial in characterizing non-uniqueness.

Definition 2.1** (Flat Extrema).**

Let $\varphi$ be any linear combination of the features $\Phi$, i.e. $\varphi\in\text{span}(\Phi)$, such that $\varphi\not\equiv 0$, $\varphi_{\max}\coloneqq\max_{s\in\mathcal{S}}\varphi(s)$ and $\varphi_{\min}\coloneqq\min_{s\in\mathcal{S}}\varphi(s)$. Let $\mathcal{N}_{\alpha}\coloneqq\left\{s\mathrel{\mathop{\mathchar 58\relax}}\varphi(s)\geq\alpha\varphi_{\max}\right\}$ for $\alpha\in[0,1]$. Then we say that $\varphi$ has a flat maximum if $\mu[\mathcal{N}_{1}]\coloneqq\int_{\mathcal{N}_{1}}\mu(s)ds>0$. Conversely, we say that $\varphi$ has a non-flat maximum if $\mu[\mathcal{N}_{1}]=0$. Similarly, we define $\mathcal{N}_{\alpha}^{-}\coloneqq\left\{s\mathrel{\mathop{\mathchar 58\relax}}\varphi(s)\leq\alpha\varphi_{\min}\right\}$ and say that $\varphi$ has a flat minimum if $\mu[\mathcal{N}_{1}^{-}]>0$ and a non-flat minimum if $\mu[\mathcal{N}_{1}^{-}]=0$.

Any linear combination of features that have flat extrema achieve their maximum (and minimum) values in regions of the state space with non-zero measure under the stationary distribution $\mu$. Conversely, linear combinations that have non-flat extrema achieve their maximum (and minimum) values in regions of the state space with non-zero measure, corresponding to sub-regions of the state space that are never visited. Figure 1 gives an idea of what functions with a flat or non-flat maximum may look like.

2.3 The $TD(\lambda)$ Operator

Tsitsiklis and Van Roy define the $TD(\lambda)$ operator $\mathcal{T}^{(\lambda)}$ in TVR (97) which we will adapt for our setting and derive a more compact form of the Bellman equation. For $\lambda\in[0,1)$, the $TD(\lambda)$ operator $\mathcal{T}^{(\lambda)}\mathrel{\mathop{\mathchar 58\relax}}L^{2}(\mathcal{S},\mu)\to L^{2}(\mathcal{S},\mu)$ is given by

[TABLE]

where $V\in L^{2}(\mathcal{S},\mu)$ and $s\in\mathcal{S}$. As Tsitsiklis and Van Roy show, the $TD(\lambda)$ operator can also be expressed as

[TABLE]

(TVR, 97, Lemma 3). We express this operator in a more compact form as

[TABLE]

where

[TABLE]

The $P_{T}^{(\lambda)}$ operator can be seen as being a geometric average over the powers of $P_{T}$. We define a $\lambda$-weighted discount factor $G$ that corresponds to the discounting performed by the $P_{T}^{(\lambda)}$ operator:

[TABLE]

Clearly $G$ is bounded in $[0,1)$. It is also important to note that for $\lambda=0$ we recover the original discount factor of $\gamma$.

As we will see later, our class of natural algorithms consists of methods that look to converge to the fixed point of a projected Bellman equation

[TABLE]

where $\Pi$ is a projection operator.

3 Projected Equation Methods

We now introduce the theory of projected equation methods and present the characteristic equation of projected Bellman equations. We note that Bertsekas similarly covers projected Bellman equation methods Ber (11). We find it useful to reiterate the concepts here as it pertains to a continuous state space and our setup.

3.1 Oblique Projection Operators

We consider the set of possible projection operators that can be applied to the Bellman equations to find an approximate solution. The projection operators that can project in any direction are collectively known as oblique projections. An oblique projection operator $\Pi\mathrel{\mathop{\mathchar 58\relax}}L^{2}(\mathcal{S},\mu)\to L^{2}(\mathcal{S},\mu)$ can be characterised as projecting onto $\text{im}(\Pi)$, the image of $\Pi$, and orthogonally to $\text{im}(\Pi^{*})$ where $\Pi^{*}$ is the adjoint operator of $\Pi$. The purpose of looking at projection operators is to find learnable, finite-dimensional representations of the value function. Thus, we will focus on oblique projection operators with finite-dimensional image. For bounded projection operators with finite-dimensional image, the image of the adjoint has the same dimension.

Proposition 3.1** (Finite-Dimensional Projections).**

Let $\Pi\mathrel{\mathop{\mathchar 58\relax}}L^{2}(\mathcal{S},\mu)\to L^{2}(\mathcal{S},\mu)$ be a bounded linear operator with finite-dimensional image and let $\Pi^{*}$ be its adjoint. Then the image of $\Pi^{*}$ has the same dimension as the image of $\Pi$.

Proposition 3.1 allows us to characterise the oblique projection operators in terms of two finite-dimensional subspaces. We will generalise slightly beyond bounded projection operators by considering the case where the image of the adjoint is still finite-dimensional but may not be of the same dimension as the original projection operator. As we discuss in the next sub-section, this will allow us to express the solution to projected Bellman equations as a system of linear equations. We define the set of projection operators that we are interested in as follows.

Definition 3.2** (Finite Rank Projection Operators).**

Let $\Phi=\{\phi_{1},...,\phi_{k}\}$ and $\Psi=\{\psi_{1},...,\psi_{n}\}$. Let $\Pi\mathrel{\mathop{\mathchar 58\relax}}L^{2}(\mathcal{S},\mu)\to L^{2}(\mathcal{S},\mu)$ be an oblique projection operator such that $\text{im}(\Pi)=\text{span}(\Phi)$ and $\text{im}(\Pi^{*})=\text{span}(\Psi)$. Then $\Pi$ can be characterised by the two sets $(\Phi,\Psi)$.

When conducting our analysis, we always assume the following.

Assumption 1**.**

Let $\Psi=\left\{\psi_{1},\ldots,\psi_{n}\right\}$ be a basis for $\text{im}(\Pi^{*})$. Also, assume that $\mathinner{\!\left\lVert\psi\right\rVert}_{1,\mu}\coloneqq\int_{\mathcal{S}}\psi_{i}(s)\mu(s)ds=1$ for all $i$.

Note that Assumption 1 results in no loss of generality. Such a basis always exists for finite-dimensional spaces. Furthermore requiring $\mathinner{\!\left\lVert\psi_{i}\right\rVert}_{1,\mu}=1$ is not restrictive. Since $\psi_{i}\in\text{im}(\Pi^{*})$ and $\text{im}(\Pi^{*})\subset L^{2}(\mathcal{S},\mu)$, we must have that $\mathinner{\!\left\lVert\psi_{i}\right\rVert}_{\mu}<\infty$, implying that $\mathinner{\!\left\lVert\psi_{i}\right\rVert}_{1,\mu}$ is also bounded and can be normalized. In the next sub-section we present the natural algorithms and how the solution to projected Bellman equations is characterised.

3.2 Natural Algorithms and the Solution to Projected Bellman Equations

We now look to determine the approximate value function $\hat{V}=\phi^{\top}w\in\text{span}(\Phi)$ found as the fixed point of a projected Bellman equation. For this task, it is natural to consider a finite rank projection operator $\Pi$ characterised by $(\Phi,\Psi)$ to find $\hat{V}$ as the fixed point of $\hat{V}=\Pi\mathcal{T}^{(\lambda)}\hat{V}$. Since the basis functions are known, the only task left is to find an expression for the parameter vector $w$. The following proposition states that $w$ is the solution to a system of linear equations.

Proposition 3.3** (Characteristic Equation for Projected Bellman Equations).**

Let $\Pi$ be a finite rank projection operator characterised by $(\Phi,\Psi)$. Suppose a unique solution exists and let $\hat{V}\in\text{span}(\Phi)$, given by $\hat{V}=\phi^{\top}w$, be the unique fixed point of the projected Bellman equation

[TABLE]

Let $A\in\mathbb{R}^{n\times k}$, $B\in\mathbb{R}^{n\times k}$, and $b\in\mathbb{R}^{n}$ be defined by

[TABLE]

respectively for $i=1,2,\ldots,n$ and $j=1,2,\ldots,k$. Then the parameter vector $w=(w_{1},\ldots,w_{k})^{\top}$ is given as the solution to the system of linear equations

[TABLE]

Proposition 3.3 suggests that any algorithm that converges to the fixed point of a projected Bellman equation in the limit has its solution characterised by the three matrix-vector quantities $A,B$, and $b$. We denote this class of algorithms as natural algorithms.

Definition 3.4** (Natural Algorithms).**

Let $\Pi$ be a finite rank projection operator characterised by $(\Phi,\Psi)$. If an algorithm converges to the fixed point $\hat{V}=\phi^{\top}w$ of a projected Bellman equation

[TABLE]

then the algorithm is a natural algorithm.

Some examples of natural algorithms are the $TD(\lambda)$, $GTD(\lambda)$ and Residual Gradient algorithms. This can be seen since both the $TD(\lambda)$ and $GTD(\lambda)$ algorithms converge to the $TD(\lambda)$ solution and the Residual Gradient algorithm was shown explicitly to solve an obliquely projected Bellman equation TVR (97); Mae (11); Sch (10).

4 A Counter-Example to Uniqueness

To motivate our results, we first present a counter-example highlighting some of the non-uniqueness issues that natural algorithms can face. Sutton and Barto provide a counter-example that highlights where Bellman-error based methods do not converge and experience non-uniqueness (SB, 18, Example 11.4). The counter-example considers the two Markov decision processes depicted in Figure 2. The edges indicate a state transition and the labels indicate the reward received. When two edges leave a single state, we assume the transition occur with equal probability. The transition matrices of MDP 1 and MDP 2 are then given by

[TABLE]

The stationary distributions are given by $\mu_{1}=(\frac{1}{3},\frac{2}{3})^{\top}$ and $\mu_{2}=(\frac{1}{3},\frac{1}{3},\frac{1}{3})^{\top}$ for MDP 1 and MDP 2 respectively. A simple linear function approximation mechanism is used with a two component parameter vector $w=(w_{1},w_{2})^{\top}$. In MDP 1, the value function can be represented exactly by $\hat{V}(1)=w_{1},~{}\hat{V}(2)=w_{2}$. In MDP 2, we assume that states 2 and 3 share a parameter value, giving $\hat{V}(1)=w_{1},~{}\hat{V}(2)=\hat{V}(3)=w_{2}$. To any RL algorithm using this feature construction, the two MDPs appear indistinguishable as the feature-reward sequence generated under the stationary distribution occur with the same probabilities. Furthermore the Bellman error, given by $E_{BE}\coloneqq\mathinner{\!\left\lVert(I-\gamma T)\Phi\hat{w}-R\right\rVert}^{2}_{\mu}$, is not a unique function of the data sample. For a parameter value $\hat{w}=0$, the Bellman error is 0 in MDP 1 whilst it is $\frac{2}{3}$ in MDP 2. This suggests that even though an algorithm minimizing the Bellman error may converge, it may converge to the wrong parameter vector.

In light of this example, we explicitly define a stronger notion of convergence to the correct solution. The next assumption asserts that there is a true environment and that the value function can be represented.

Assumption 2**.**

Let $R^{*},T^{*}$ be the true environment and assume that there exists a parameter vector $w^{*}$ such that the value function $V^{*}$ can be represented as $V^{*}(s)=\phi^{\top}(s)w^{*}$.

We now define convergence as follows.

Definition 4.1**.**

(Convergence) An algorithm is said to converge if it converges to $w^{*}$ or, equivalently, $V^{*}$.

5 Main Results

We now present our main results and discuss their implications. Our main theorem directly characterises convergence in terms of a property on the features.

Theorem 5.1** (Flatness Condition on Features).**

Natural algorithms converge if and only if all non-zero linear combinations of the features $\Phi$ have flat extrema.

It is important to note that Theorem 5.1 holds for all finite rank projections onto $\text{span}(\Phi)$. The factor determining convergence is the choice of features. Theorem 5.1 provides a restrictive condition on the possible feature choices available for natural algorithms with linear function approximation to converge. Not only are the usual assumptions of linear independence in the features necessary; it is required that all linear combinations of the features have flat extrema.

An immediate consequence of Theorem 5.1 is that state aggregation methods are always safe feature construction choices. If states are aggregated such that each subset has non-zero measure under the stationary measure, all value functions in the span of these features have flat extrema. An example of this is a partitioning-based state aggregation scheme, shown in Figure 3, that visibly has flat extrema. If there exist aggregated states with measure zero, these states would be unobserved under the stationary measure and would not impact the representation. This result summarised in the following corollary.

Corollary 5.2** (State Aggregation).**

State aggregation is a safe feature construction choice for natural algorithms.

Though state aggregation is a sufficient choice for natural algorithms to avoid convergence issues, algorithms with other feature constructions have been shown to converge TVR (96). We present a condition in the next sub-section to help determine and construct convergent natural algorithms.

5.1 A Projection Perspective

Under Assumption 1, the inner product between $\psi_{i}$ and any function $\varphi\in\text{span}(\Phi)$ can be seen as the projection of $\varphi$ onto $\psi_{i}$. Our next result presents a condition on these projections which can aid in determining convergent algorithms with feature constructions other than state aggregation.

Theorem 5.3** (Convergent Natural Algorithms).**

All natural algorithms characterized by $(\Phi,\Psi)$ converge if and only if there exists an $i$ such that for all $\varphi\in\text{span}(\Phi)$

[TABLE]

Theorem 5.3 guarantees a natural algorithm’s convergence if it can project the extremal regions of any approximate value function. A simple case is when $\langle\psi_{i},\varphi\rangle_{\mu}=\varphi_{\max}$ or $\langle\psi_{i},\varphi\rangle_{\mu}=\varphi_{\min}$. This occurs precisely when $\varphi$ has flat extrema and $\psi_{i}$ projects $\varphi$ on the sub-regions of the state space where $\varphi$ achieves its extremes. An example of this is shown in Figure 4.

Example 1**.**

We now provide an explicit example of how Theorem 5.3 can help construct convergent natural algorithms. Consider the piece-wise linear features displayed in Figure 5. Any linear combination of these features also results in piece-wise linear functions that have flat maxima. Then any natural algorithm which has the functions $\psi_{i}$ positive on the regions of the state space where the features achieve their flat maxima will satisfy Theorem 5.3. Effectively, such an algorithm disregards any information about the regions of the state space that are not in the flat maxima of the features. Such an algorithm can be determined without knowledge of the value function since it only depends on the features, which are chosen apriori.

Example 2**.**

An example of a convergent natural algorithm that projects on the states that achieve the maximum value is the modified value iteration approach presented by Tsitsiklis and Van Roy TVR (96). At the outset, $K$ representative states $s_{1},\ldots,s_{K}\in\mathcal{S}$ are chosen and their feature vectors $\phi(s_{1}),\ldots,\phi(s_{K})$ are constructed. The remaining states are then chosen from within the convex hull of the feature vectors of the representative states. In this manner, the feature construction ensures that the maximum value of all approximate value functions are centred on the representative states. The modified value iteration then solves for the fixed point of

[TABLE]

where $\Phi^{\dagger}$ is the left inverse of $\Phi$. In this construction, the projection operator is given by $\Phi\Phi^{\dagger}$. Since the non-representative states are composed from the representative states, the algorithm proceeds by computing only on the representative states. Thus, this method effectively takes a projection on the points of the state space that achieve the maximum value.

6 Framework of Analysis

In this section we establish the framework we use for analysing convergence and non-uniqueness for natural algorithms. We call our framework the Bellman template. We specifically look to capture two phenomena of non-uniqueness that were displayed in Sutton and Barto’s counter-example: how natural algorithms may converge to the wrong solution even when the value function is representable and how different MDP environments with different optimal parameter vectors appear indistinguishable under projection. We formally define the Bellman Template as follows.

Definition 6.1** (Bellman Template).**

Let $\Pi$ be a finite rank projection characterised by $(\Phi,\Psi)$. Let $w\in\mathbb{R}^{k}$, $R\mathrel{\mathop{\mathchar 58\relax}}\mathcal{S}\to\mathbb{R}$ such that $\mathinner{\!\left\lVert R\right\rVert}_{\infty}<\infty$, and $T\mathrel{\mathop{\mathchar 58\relax}}\mathcal{S}\times\mathcal{S}\to[0,1]$ such that $\int_{\mathcal{S}}T(s^{\prime}|s)ds^{\prime}=1$. The following constraints on $(w,R,T)$ are collectively defined as the Bellman template:

• •

$(w,R,T)$ satisfy the Bellman equation

[TABLE]

where $\hat{V}(s)=\phi^{\top}(s)w$.

• •

Let $A\in\mathbb{R}^{n\times k}$, $B\in\mathbb{R}^{n\times k}$ and $b\in\mathbb{R}^{n}$. $R$ and $T$ satisfy

[TABLE]

for $i=1,\ldots,n$ and $j=1,\ldots,k$ respectively and $w$ satisfies

[TABLE]

We say that a triple $(w,R,T)$ is a solution to the Bellman template if it satisfies these constraints.

A Bellman template solution represents an MDP environment (through the expected reward function $R$ and the transition density function $T$) and its value function under the stationary policy (through the parameter vector $w$). The first constraint in the Bellman template restricts our attention to the case where the value function is exactly representable by the chosen features $\Phi$. The second constraint provides a condition to capture when different solutions to the Bellman template appear indistinguishable under projection. In particular, we consider when different solutions $(w,R,T)$ produce the same quantities $A,B$, and $b$ that characterize solutions. It may seem strange that non-uniqueness could present an issue since a solution to a projected Bellman equation is uniquely determined by $A,B$ and $b$. The crucial difference however is that we now let the environment variables, $R$ and $T$, vary.

We define the condition of ambiguity, which represents non-uniqueness, as follows.

Definition 6.2** (Ambiguity).**

Ambiguity holds if the Bellman template has more than one $(w,R,T)$ solution that have different parameters $w$.

Under ambiguity, different MDP environments with different optimal value functions appear the same to natural algorithms under projection. Therefore, natural algorithms fail to converge under ambiguity.

7 Theorem 5.1 Proof Idea

For brevity, we only present the key ideas behind the proof of Theorem 5.1 here. The full proof however can be found in Appendix B. We first present a supporting result that characterises ambiguity.

Theorem 7.1**.**

Let $0\not\equiv\varphi\in\text{span}(\Phi)$ and $\varphi_{\min}\coloneqq\min\limits_{s\in\mathcal{S}}\varphi(s)$ and $\varphi_{\max}\coloneqq\max\limits_{s\in\mathcal{S}}\varphi(s)$. Ambiguity holds if and only if there exists an $f\mathrel{\mathop{\mathchar 58\relax}}\mathcal{S}\to[\varphi_{\min},\varphi_{\max}]$ such that

[TABLE]

for all $i=1,\ldots,n$, where $\chi_{i}(s)=\psi_{i}(s)\mu(s)$.

The idea behind the proof of Theorem 5.1 is to construct a function $f$ that satisfies Theorem 7.1. We show that such a function $f$ exists if and only if there exists a non-zero linear combination of the features $\varphi$ that has non-flat extrema. Then by the equivalence given in Theorem 7.1, ambiguity holds meaning natural algorithms will fail to converge. Taking the contrapositive then gives Theorem 5.1.

To construct a suitable function, first consider the function $\tilde{f}\coloneqq\frac{1}{G}\varphi$. Clearly $\tilde{f}$ satisfies (4). For $\varphi_{\max}<0$, $\tilde{f}$ satisfies the upper bound on the range as $\tilde{f}(s)\leq\varphi_{\max}$ for all $s\in\mathcal{S}$. Similarly, in the case where $\varphi_{\min}>0$, $\tilde{f}$ satisfies the lower bound on the range. However when $\varphi_{\max}>0$, $\tilde{f}$ exceeds the upper bound. Again in similar fashion, when $\varphi_{\min}<0$, $\tilde{f}$ exceeds the lower bound. We now look to construct a function from $\tilde{f}$ that does not exceed the bounds in these cases whilst still satisfying (4). We will focus on the $\varphi_{\max}>0$ case, noting that $\varphi_{\min}<0$ is treated the same way. We consider capping $\tilde{f}$ at $G\varphi_{\max}$ and spreading the ‘cut’ pinnacle across the basis functions $\chi_{i}$ in a way that satisfies (4). Consider

[TABLE]

where $\delta(s)\coloneqq\max\{0,\frac{1}{G}\varphi(s)-G\varphi_{\max}\}$ is the cut pinnacle. Now define $f\coloneqq\bar{f}+g$, where $g$ is some function. We look to find $g$ as the pinnacle $\delta$ projected onto the basis functions $\chi_{i}$ in such a way that (4) is satisfied. It can be shown that (4) is equivalent to

[TABLE]

for all $i=1,\ldots,n$. Thus there are $n$ constraints for $g$ to satisfy. To get a gauge on whether spreading $\delta$ in this fashion may be possible, consider the following. The height of the portion of $\frac{1}{G}\varphi$ that exceeds $\varphi_{\max}$ is of order O$(1-G)$. Also, if $\varphi$ has a non-flat maximum and $G$ is sufficiently close to 1, this portion has ‘mass’ o$(1-G)$ since $\mu\left[\mathcal{N}_{G}\right]$ goes to 0 as $G$ goes to 1. As $g$ spreads the mass of $\delta$ over $\chi_{i}$ independently from $G$, it is of order o$(1-G)$. Thus for $G$ sufficiently close to 1, $g(s)<\delta(s)$. Thus it seems plausible that $f=\bar{f}+g=\frac{1}{G}\varphi-\delta+g$ would not surpass the upper bound of $\varphi_{\max}$. We employ an analogous construction to find a function $f^{-}$ that satisfies the lower bound for the case where $\varphi_{\min}<0$. Finally, we are able to combine the different functions we construct to satisfy the range conditions for each case in Theorem 7.1 and ultimately show our result. Figure 6 provides a sketch of the construction we employ in this proof.

8 Conclusion

We have established that natural algorithms are inherently prone to fail without careful consideration of the choice of features. In particular, natural algorithms converge if and only if the features chosen only allow for linear combinations with flat extrema. We also presented a condition from a projection perspective that can help determine the convergence of natural algorithms as well. Given our results, we justify that state aggregation features are sufficient for all natural algorithms to converge. We also provide a condition under which natural algorithms with other feature constructions can converge if they project upon the extreme regions of the features. In doing so, we show how a convergent natural algorithm can be constructed from our result as well as arguing for the convergence of the modified value iteration approach presented in TVR (96).

It is important to note that our results begin where the assumptions in previous convergence proofs do not hold. As an example, $TD(\lambda)$ is known to converge on-policy but counter-examples exist for the off-policy case TVR (97). In our analysis, the off-policy case is subsumed by the projection operators we consider as we do not restrict them to be non-expansions with respect to the $\mathinner{\!\left\lVert\cdot\right\rVert}_{\mu}$-norm. Thus, our result also implies the divergence of Q-learning.

We note that the natural algorithm class, while extensive, does not cover all known RL algorithms with linear function approximation. In particular the ETD algorithm, introduced in SMW (16), does not fall within our natural algorithm class. The ETD algorithm includes an extra interest function $i$ that alters the visiting probabilities of states, meaning we are no longer working with the stationary distribution $\mu$.

An important factor in determining whether our results will hold in practice is the choice of discount factor. Throughout our analysis, the discount factor plays an important role in defining the extrema regions. As the discount factor moves away from one and towards zero, it becomes less likely that the non-flat extrema property will occur. Thus the discount factor determines the degree to which feature choices that deviate from flat extrema allow natural algorithms to converge. Also, our analysis centres on a strict notion of convergence to the true value function. Investigating whether our analysis can extend to characterise non-uniqueness when considering approximate value functions is an interesting open question.

Appendix A Supporting Results

In this section we present the supporting results we use to prove Theorem 5.1 and Theorem 5.3 as well as any omitted proofs here.

The first lemma places ambiguity in relation to a non-trivial null-space of the linear system of equations that characterize solutions to projected Bellman equations. A non-trivial null-space is exactly what governs non-uniqueness in a system of linear equations; the only added difficulty is the difference that varying environment variables $R$ and $T$ presents.

Lemma A.1**.**

Ambiguity holds if and only if there exists $0\not\equiv v\in\mathbb{R}^{k}$ such that $\left(A-\gamma B\right)v=0$.

Proof.

Under ambiguity, there exists another solution to the Bellman template $(w^{0},R^{0},T^{0})$ such that $w^{0}\neq w^{*}$. From the derivation of Proposition 3.3, we can show that both $w=w^{0}$ and $w=w^{*}$ satisfy

[TABLE]

Then trivially, $(A-\gamma B)v=0$ holds since we can take $v=w^{*}-w^{0}$. Now consider the reverse. Assume that $(A-\gamma B)v=0$ holds for some $v\neq 0$. For $\xi>0$, let us define

[TABLE]

Then $(w^{\xi},R^{\xi},T^{\xi})$ satisfies the generalised Bellman equation since

[TABLE]

which is precisely the left-hand side of the generalised Bellman equation. Let $A^{\xi}\in\mathbb{R}^{n\times k}$, $B^{\xi}\in\mathbb{R}^{n\times k}$ and $b\in\mathbb{R}^{n}$ be given by

[TABLE]

for $i=1,\ldots,n$ and $j=1,\ldots,k$. We trivially have that $A^{\xi}=A$ and we note that $B^{\xi}=B$ since $T^{\xi}=T^{*}$. By Proposition 3.3, applying the projection $\Pi$ characterised by $(\Phi,\Psi)$ onto the generalised Bellman equation induces the following

[TABLE]

We then have the following derivation

[TABLE]

where (a) follows since $(A-\gamma B)v=0$. Thus we have that $b^{\xi}=b$ as well. Thus, for any $\xi>0$, $(w^{\xi},R^{\xi},T^{\xi})$ satisfies the generalised Bellman equation as well and so ambiguity holds. ∎

The next corollary is effectively a restatement of Lemma A.1 that will be easier to work with later.

Corollary A.2**.**

For all $i=1,\ldots,n$, let $\chi_{i}(s)\coloneqq\psi_{i}(s)\mu(s)$ where $\mu$ is the stationary distribution. Then ambiguity holds if and only if there exists $T$ and $0\not\equiv\varphi\in\text{span}(\Phi)$ such that for all $i=1,\ldots,n$,

[TABLE]

Proof.

From Theorem A.1, we have that ambiguity holds if and only if there exists $0\not\equiv v\in\mathbb{R}^{k}$ such that

[TABLE]

Now for all $i=1,\ldots,n$ and $j=1,\ldots,k$

[TABLE]

for some $T$. Now note that $A_{i\cdot}$ and $B_{i\cdot}$ are row vectors of $A$ and $B$ for all $i=1,\ldots,n$. Then since $\left(A-\gamma B\right)v=0$, we have for all $i=1,\ldots,n$ the following derivation

[TABLE]

Here in (a) we substituted in the derivation of $A_{ij}-\gamma B_{ij}$ from above and in (b) we used the Fubini-Tonelli theorem to swap the sum and the integral. Since we’ve only looked at equivalences, our if and only if result holds. ∎

The next lemma presents a useful characterisation of non-flat extrema as the $\lambda$-weighted discount factor approaches the limit.

Lemma A.3**.**

Let $0\not\equiv\varphi\in\text{span}(\Phi)$. Then $\varphi$ has non-flat maximum if and only if $\mu[\mathcal{N}_{G}]\to 0$ as $G\to 1$. Similarly, $\varphi$ has non-flat minimum if and only if $\mu[\mathcal{N}_{G}^{-}]\to 0$ as $G\to 1$.

Proof.

We will only prove that $\varphi$ has non-flat maximum if and only if $\mu[\mathcal{N}_{G}]\to 0$ as $G\to 1$ noting that adapting the proof for the non-flat minimum case is routine.

We first show the forward direction. Since $\varphi$ has a non-flat maximum, we have that $\mu[\mathcal{N}_{1}]=0$. Also $\mu[\mathcal{S}]$ can be written as

[TABLE]

Re-arranging and taking the limit as $G$ goes to 1 gives

[TABLE]

Thus the forward direction holds. Now consider the reverse direction. Suppose $\mu[\mathcal{N}_{G}]\to 0$ as $G\to 1$. Then trivially $\lim_{G\to 1}\mu[\mathcal{N}_{G}]=\mu[\mathcal{N}_{1}]=0$. Thus the result holds. ∎

The next result shows that $P^{(\lambda)}_{T}$ is a non-expansion.

Proposition A.4**.**

The operator $P^{(\lambda)}_{T}$ is a non-expansion with respect to $\mathinner{\!\left\lVert\cdot\right\rVert}_{\mu}$.

Proof.

We first show that $P_{T}$ is a non-expansion before showing the result. To see that $P_{T}$ is a non-expansion note that

[TABLE]

Here (a) follows by Jensen’s inequality, (b) follows by the Tonelli-Fubini theorem, and (d) follows since $\mu$ is the stationary distribution. Since the quadratic function is monotonically increasing on $\mathbb{R}_{+}$, we have that $\mathinner{\!\left\lVert P_{T}V\right\rVert}_{\mu}\leq\mathinner{\!\left\lVert V\right\rVert}_{\mu}$ and so $P_{T}$ is non-expansive.

Now to see that $P^{(\lambda)}_{T}$ is non-expansive, we have the following derivation:

[TABLE]

Here (a) follows as $P_{T}$ is a non-expansion and (b) follows since $\frac{1-\lambda}{1-\gamma\lambda}\leq 1$ as $\lambda\geq\gamma\lambda$. Thus $P^{(\lambda)}_{T}$ is a non-expansion. ∎

Appendix B Omitted Proofs

Proposition 3.1 Proof

Proof.

Let $l_{y}(x)=\langle\Pi x,y\rangle_{\mu}$. Then by the Riesz representation theorem, there exists $z_{y}$ such that $l_{y}(x)=\langle x,z_{y}\rangle_{\mu}$ and by definition, $\Pi^{*}y=z_{y}$. Now note that if $y\in\text{im}(\Pi)^{\perp}$, then $y\in\text{ker}(\Pi^{*})$ and so $z_{y}=0$. Now let $z\in L^{2}(\mathcal{S},\mu)$. Since $L^{2}(\mathcal{S},\mu)$ can be decomposed into the direct sum of $\text{im}(\Pi)$ and $\text{im}(\Pi)^{\perp}$, then there exists $z_{1}\in\text{im}(\Pi)$ and $z_{2}\in\text{im}(\Pi)^{\perp}$ such that $z=z_{1}+z_{2}$. Applying $\Pi^{*}$ to $z$ then gives

[TABLE]

Thus since $z_{1}\in\text{im}(\Pi)$, which is finite-dimensional, the image of $\Pi^{*}$ must also have the same dimensions. ∎

Proposition 3.3 Proof

Proof.

Since $\Pi$ projects orthogonally to $\text{im}(\Pi^{*})$ and onto $\text{im}(\Pi)$, we have that $\mathcal{T}\hat{V}-\Pi\mathcal{T}\hat{V}\perp_{\mu}\text{im}(\Pi^{*})$. So for all $i=1,\ldots,n,~{}\langle\psi_{i},\mathcal{T}\hat{V}-\Pi\mathcal{T}\hat{V}\rangle_{\mu}=0$. Substituting $\hat{V}$ for $\Pi\mathcal{T}\hat{V}$, expanding and re-arranging gives

[TABLE]

Re-arranging now gives

[TABLE]

Then given our definitions of $A,B$, and $b$, we have a linear system of $k$ equations given by $\left(A-\gamma B\right)w=b$. ∎

Theorem 7.1 Proof

Proof.

By Corollary A.2, ambiguity holds if and only if there exists a$T$ and $0\not\equiv\varphi\in\text{span}(\Phi)$ such that for all $i$

[TABLE]

We define $\tilde{P}_{T}^{(\lambda)}$ as

[TABLE]

The $\tilde{P}_{T}^{(\lambda)}$ operator can be viewed as a re-normalised version of $P_{T}^{(\lambda)}$. It can then be seen that $\tilde{P}_{T}^{(\lambda)}\varphi(s)$ is bound between $\varphi_{\min}$ and $\varphi_{\max}$ since

[TABLE]

and

[TABLE]

Now (5) can be re-written as

[TABLE]

So in the forward direction, we can simply take $f(s)=\tilde{P}^{(\lambda)}_{T}\varphi(s)$. Now in the reverse direction, we note that any function $f\mathrel{\mathop{\mathchar 58\relax}}\mathcal{S}\to[\varphi_{\min},\varphi_{\max}]$ can be represented as $\tilde{P}^{(\lambda)}_{T}\varphi(s)$ for some $T$. Thus the result holds. ∎

Theorem 5.1 Proof

We look to construct a function $f$ that satisfies Theorem 7.1 as well as

[TABLE]

for $i=1,\ldots,n$. We show that such a function $f$ exists in each case if and only if there exists $0\not\equiv\varphi\in\text{span}(\Phi)$ that has non-flat extrema. Then by the equivalence given in Theorem 7.1, ambiguity holds. Taking the contrapositive then gives Theorem 5.1. We first look to find a function $f$ that satisfies (6) and the upper bound on the range.

Consider $\tilde{f}(s)=\frac{1}{G}\varphi(s)$. For $\varphi_{\max}<0$, $\tilde{f}$ clearly satisfies (6) and the upper bound on the range; but for $\varphi_{\max}>0$, $\tilde{f}$ exceeds the upper bound. Instead, we look to construct a function from $\tilde{f}$ whereby we ‘cut’ off the portion that exceeds the upper bound and project it across the functions $\chi_{i}$ for $i=1,\ldots,n$ to satisfy the upper bound whilst still satisfying (6). Consider

[TABLE]

where $\delta(s)\coloneqq\max\{0,\frac{1}{G}\varphi(s)-G\varphi_{\max}\}$ is the cut pinnacle. By construction, $\bar{f}$ satisfies the upper bound. Now let us define

[TABLE]

We look to find $g$ such that $f$ satisfies (6) and the upper bound. The following derivation derives a linear system of equations that constrain the function $g$ such that $f$ satisfies (6). Starting from (6), we have

[TABLE]

Here (a) follows by the definition of $f$ and $\bar{f}$. Cancelling out $\int_{S}\chi_{i}(s)\varphi(s)ds$ and $G$ from both sides of the equation gives

[TABLE]

We now note that the functions $\chi_{1},\ldots,\chi_{n}$ forms a linearly independent set. To see this, suppose that for all $s\in\mathcal{S}$

[TABLE]

and $b_{1},\ldots b_{n}$ are not all equal to 0. Then since $\chi_{i}(s)=\mu(s)\psi_{i}(s)$, and $\mu(s)>0$, we must have

[TABLE]

This is a contradiction since the set of functions $\Psi=\{\psi_{1},\ldots,\psi_{n}\}$ is a linearly independent set. Thus, let $g$ be given by

[TABLE]

where $a_{i}\in\mathbb{R}$ for all $i$. Then from (7) we have

[TABLE]

where in (a) we swapped the summation and the integrand by the Fubini-Tonelli theorem. Now as a notational shorthand, let $\langle f,g\rangle\coloneqq\int_{\mathcal{S}}f(s)g(s)ds$. Then we have

[TABLE]

Let $X\in\mathbb{R}^{n\times n}$ and $\bar{\delta}\in\mathbb{R}^{n}$ be defined by

[TABLE]

respectively. Together $a$, X and $\bar{\delta}$ form a system of linear equations given by

[TABLE]

Note that since $\chi_{1},\ldots,\chi_{n}$ is a set of linearly independent functions, $X$ is full rank and thus invertible. We can express $a$ as

[TABLE]

Let $\chi(s)=\left(\chi_{1}(s),\ldots,\chi_{n}(s)\right)$. We can now express $g$ as

[TABLE]

We now explicitly define three infinity norms we will use to bound $g$. For a function $f(s)=(f_{1}(s),\ldots,f_{n}(s))$, vector $u\in\mathbb{R}^{n}$, and matrix $A\in\mathbb{R}^{n\times n}$, the norms are given by

[TABLE]

Under these norm definitions, we see that $\mathinner{\!\left\lVert X^{-1}\right\rVert}_{\infty}<\infty$ since $X$ is invertible. The function $\chi$ has its infinity norm given by $\mathinner{\!\left\lVert\chi\right\rVert}_{\infty}=\max_{1\leq i\leq n}\sup_{s\in\mathcal{S}}\mathinner{\!\left\lvert\chi_{i}(s)\right\rvert}$. To see that this is finitely bounded, recall that for any $i$, $\int_{\mathcal{S}}\chi_{i}(s)ds=1$. We can then split the $\chi_{i}$ into two functions $\chi_{i}^{+}$ and $\chi_{i}^{-}$ where $\chi_{i}^{+}$ is the same value as $\chi_{i}$ when it is positive and $\chi_{i}^{-}$ is the same value as $\chi_{i}$ when it is negative. Then since

[TABLE]

it must be the case that both individual integrals are finite. Thus $\mathinner{\!\left\lvert\chi_{i}(s)\right\rvert}<\infty$ for all $i$ and $s$. We now look to derive a bound for $\bar{\delta}$. Consider the set $\mathcal{N}_{G^{2}}$ given by

[TABLE]

Note that $\delta(s)>0$ if and only if $s\in\mathcal{N}_{G^{2}}$. Let $\mathbf{1}_{\mathcal{N}_{G^{2}}}$ be the characteristic function for $\mathcal{N}_{G^{2}}$. Then for all $i=1,\ldots,n$, we have the following derivation

[TABLE]

Here (a) follows by definition of $\chi_{i}$ and (b) follows by definition of $\delta$. In (c), we let $s_{\max}$ denote the value of $s\in\mathcal{S}$ that $\psi_{i}$ achieves a maximum. Finally, (d) follows by definition of $\mu\left[\mathcal{N}_{G^{2}}\right]$. Thus, $\bar{\delta}$ can be bounded by

[TABLE]

Combining the bounded quantities, we have that $g$ is bounded by

[TABLE]

where to simplify notation we let $C$ be the constant defined as $C=\mathinner{\!\left\lVert\chi\right\rVert}_{\infty}\mathinner{\!\left\lVert X^{-1}\right\rVert}_{\infty}\psi_{i}(s_{\max})$. Note that by definition, $\bar{f}$ is bounded by $G\varphi_{\max}$. As a result, we can now bound $f$ from above by

[TABLE]

Now $\varphi_{\max}\left(G+C\left(\frac{1}{G}-G\right)\mu\left[\mathcal{N}_{G^{2}}\right]\right)\leq\varphi_{\max}$ if and only if

[TABLE]

This preceding inequality holds if and only if

[TABLE]

which is equivalent to

[TABLE]

Since $\varphi$ has a non-flat maximum, by Lemma A.3 we have that as $G\to 1$

[TABLE]

whilst we have as $G\to 1$,

[TABLE]

Thus, $\mu\left[\mathcal{N}_{G^{2}}\right]\leq\frac{G}{C\left(1+G\right)}$ if $\varphi$ has non-flat maximum. Thus $f(s)\leq\varphi_{\max}$ for all $s\in\mathcal{S}$ and $f$ satisfies the upper bound.

By an analogous argument to the above, we can also find a construction to satisfy the lower bound in the different cases and (6). For $\varphi_{\min}>0$, $f=\frac{1}{G}\varphi(s)$ satisfies (6) and the lower bound. For $\varphi_{\min}<0$, we can find a function $f^{-}$ that satisfies (6) and has range greater than the lower bound given $\varphi$ has non-flat minimum. We define $f^{-}$ by

[TABLE]

where $g^{-}$ is some function chosen such that $f^{-}$ satisfies (6). The function $\tilde{f}(s)$ is given by

[TABLE]

where $\delta^{-}(s)$ is given by

[TABLE]

By construction, $\tilde{f}$ satisfies the lower bound. We now look to derive a system of linear equations to constrain $g^{-}$ such that (6) is satisfied. Starting from (6) we have

[TABLE]

Cancelling out from both sides $\int_{\mathcal{S}}\chi_{i}(s)\varphi(s)$ ds and re-arranging gives

[TABLE]

We let $g^{-}$ be given by

[TABLE]

Let $\bar{\delta}^{-}\in\mathbb{R}^{n}$ be given by

[TABLE]

Then $g^{-}$ is given by

[TABLE]

We now look to bound $\bar{\delta}^{-}$. Let $\mathcal{N}^{-}_{G^{2}}\coloneqq\{s\in\mathcal{S}\mathrel{\mathop{\mathchar 58\relax}}\varphi(s)\leq G^{2}\varphi_{\min}\}$. For all $i=1,\ldots,n$ we have the following derivation

[TABLE]

In (a), we used the fact that $\delta^{-}(s)$ is only greater than 0 for $s\in\mathcal{N}^{-}_{G^{2}}$ and that it is upper bounded by $\varphi_{\min}\left(G-\frac{1}{G}\right)$. In (b) we define $s_{\min}$ as the value of $s\in\mathcal{S}$ where $\psi_{i}$ achieves a minimum. Finally, (c) follows by definition of $\mu\left[\mathcal{N}^{-}_{G^{2}}\right]$. Having bound $\int_{\mathcal{S}}\chi_{i}(s)\delta^{-}(s)ds$ for all $i$, we have that $\bar{\delta}^{-}$ is bound in the infinity norm

[TABLE]

Thus, $g^{-}(s)$ is bounded in the infinity norm by

[TABLE]

where to simplify notation we define the constant $D\coloneqq\mathinner{\!\left\lVert\chi\right\rVert}_{\infty}\mathinner{\!\left\lVert X^{-1}\right\rVert}_{\infty}\psi_{i}(s_{\min})$. Given that $\tilde{f}(s)$ is bound below by $G\varphi_{\min}$, we have that $f^{-}(s)$ is bounded as follows.

[TABLE]

Now $G\varphi_{\min}-D\varphi_{\min}\left(G-\frac{1}{G}\right)\mu\left[\mathcal{N}^{-}_{G^{2}}\right]\geq\varphi_{\min}$ if and only if

[TABLE]

since $\varphi_{\min}<0$. Re-arranging shows that this inequality holds if and only if

[TABLE]

Since $\varphi$ has non-flat minimum, we have that $\mu\left[\mathcal{N}^{-}_{G^{2}}\right]\to 0$ as $G\to 1$ whereas $\frac{G}{D\left(1+G\right)}\to\frac{1}{2D}$. Thus, the inequality holds if $\varphi$ has non-flat minimum.

Note that $-\delta(s)+g(s)$ is only non-zero on $\mathcal{N}_{G^{2}}$ and $\delta^{-}(s)-g^{-}(s)$ is only non-zero on $\mathcal{N}^{-}_{G^{2}}$ and the two sets do not intersect. Thus, combining the functions $f$ and $f^{-}$ we have

[TABLE]

that satisfies the range and (4) if $\varphi$ has non-flat extrema. Thus, ambiguity holds if and only if there exists a $0\not\equiv\varphi\in\text{span}(\Phi)$ with non-flat extrema. Taking the contrapositive proves Theorem 5.1.

Theorem 5.3 Proof

By Corollary A.2, we have that ambiguity holds if and only if there exists a $T$ and $0\not\equiv\varphi\in\text{span}(\Phi)$ such that for $i=1,\ldots,n$, $\int_{\mathcal{S}}\chi_{i}(s)\varphi(s)ds=\gamma\int_{\mathcal{S}}\chi_{i}(s)P_{T}^{(\lambda)}\varphi(s)ds$. By expanding $P_{T}^{(\lambda)}$ we get

[TABLE]

The function $\varphi$ is bounded between $\varphi_{\min}$ and $\varphi_{\max}$. Furthermore, we have that $P_{T}\varphi_{\max}=\varphi_{\max}$ and $P_{T}\varphi_{\min}=\varphi_{\min}$. Thus for any $k>0$ the following holds:

[TABLE]

Furthermore, we note that

[TABLE]

by the geometric series. Then using the inequality in 9, $P_{T}^{(\lambda)}$ is bounded by:

[TABLE]

Now note that

[TABLE]

where the last equality holds due to Assumption 1. Similarly, $\int_{\mathcal{S}}\chi_{i}(s)\varphi_{\max}(s)ds=\varphi_{\max}$. Thus the following holds:

[TABLE]

Now we note that,

[TABLE]

Here (a) follows by Corollary A.2 and (b) follows by definition of $\chi_{i}$. Substituting in $\langle\psi_{i},\varphi\rangle_{\mu}$ give us that ambiguity holds if and only if there exists $0\not\equiv\varphi\in\text{span}(\Phi)$ such that for all $i$,

[TABLE]

Taking the contrapositive of this statement gives the desired result.

Appendix C List of Notation

In the following, let $\mathcal{H}$ and $\mathcal{W}$ be two vector spaces and $A\mathrel{\mathop{\mathchar 58\relax}}\mathcal{H}\to\mathcal{W}$ a linear transformation.

Notation

• RL

Reinforcement Learning.

• MDP

Markov Decision Process.

• $TD$

Temporal difference.

• $\mathcal{S}$

The state space.

• $\gamma$

The discount factor, i.e. $\gamma\in[0,1)$.

• $G$

The $\lambda$-weighted discount factor given by $G\coloneqq\frac{(1-\lambda)\gamma}{1-\lambda\gamma}$.

• $G\to 1$

$G$ approaches a value of 1.

• $\mu$

The stationary distribution.

• $\mathcal{T}$

The Bellman operator.

• $\Phi$

The set of chosen features: $\Phi\coloneqq\{\phi_{1},\phi_{2},\ldots,\phi_{k}\}$, $k\in\mathbb{N}$.

• $\phi(s)$

A feature vector for a given state $s\in\mathcal{S}$: $\phi(s)=(\phi_{1}(s),\ldots,\phi_{k}(s))^{\top}$.

• $\varphi_{\min}$

$\varphi_{\min}\coloneqq\min_{s\in\mathcal{S}}\varphi(s)$.

• $\varphi_{\max}$

$\varphi_{\max}\coloneqq\max_{s\in\mathcal{S}}\varphi(s)$.

• $T$

The transition density function.

• $P_{T}$

An operator such that for a function $f\mathrel{\mathop{\mathchar 58\relax}}\mathcal{S}\to\mathbb{R}$, $(P_{T}f)(s)\coloneqq\int_{\mathcal{S}}T(s^{\prime}|s)f(s^{\prime})ds^{\prime}$.

• $P_{T}^{(\lambda)}$

An operator defined as $P_{T}^{(\lambda)}\coloneqq(1-\lambda)\gamma\sum_{m=0}^{\infty}(\lambda\gamma)^{m}(P_{T})^{m+1}$.

• $\langle f,g\rangle_{\mu}$

An inner product defined as $\langle f,g\rangle_{\mu}\coloneqq\int_{\mathcal{S}}f(s)g(s)\mu(s)ds$.

• $\mathinner{\!\left\lVert f\right\rVert}_{\mu}$

The norm defined by the inner product $\langle f,g\rangle_{\mu}$, i.e. $\mathinner{\!\left\lVert f\right\rVert}_{\mu}\coloneqq\sqrt{\langle f,f\rangle_{\mu}}$.

• $\mathinner{\!\left\lVert f\right\rVert}_{1,\mu}$

$\mathinner{\!\left\lVert f\right\rVert}_{1,\mu}\coloneqq\int_{\mathcal{S}}f(s)\mu(s)ds$.

• $\mathbb{R}$

The set of real numbers.

• $\text{im}(A)$

The image of $A$, that is the set $\text{im}(A)\coloneqq\{Ax\mathrel{\mathop{\mathchar 58\relax}}x\in\mathcal{H}\}$.

• $\text{ker}(A)$

The kernel of $A$, that is the set $\text{ker}(A)\coloneqq\{x\in\mathcal{H}\mathrel{\mathop{\mathchar 58\relax}}Ax=0\}$.

• $\text{span}(B)$

The span of $B$.

• $\text{dim}(\mathcal{H})$

The dimension of $\mathcal{H}$.

• $x\perp y$

x is perpendicular to y.

• $x\perp_{\mu}y$

Given an inner product $\langle\cdot,\cdot\rangle_{\mu}$, this notation denotes that $x$ is perpendicular to $y$ with respect to $\langle\cdot,\cdot\rangle_{\mu}$.

• $\mu\left[\mathbb{R}\right]$

$\mu\left[\mathbb{R}\right]\coloneqq\int_{\mathbb{R}}\mu(s)ds$.

• $\mathbf{1}_{B}(s)$

The characteristic function for the set $B$.

• $\mathbf{1}(s^{\prime}=s)$

The indicator function.