Why Target Networks Stabilise Temporal Difference Methods

TL;DR

This paper provides a theoretical explanation for why target networks stabilize temporal difference learning in deep reinforcement learning, showing they mitigate poor conditioning and can guarantee convergence under certain conditions.

Contribution

It formalizes the concept of partially fitted policy evaluation, characterizes the deadly triad, and explains how target networks improve stability and convergence in complex RL settings.

Findings

Target networks mitigate poor conditioning in TD updates.

Proper tuning of target network update frequency ensures convergence.

The framework bridges fitted methods and semigradient TD algorithms.

Abstract

Integral to recent successes in deep reinforcement learning has been a class of temporal difference methods that use infrequently updated target values for policy evaluation in a Markov Decision Process. Yet a complete theoretical explanation for the effectiveness of target networks remains elusive. In this work, we provide an analysis of this popular class of algorithms, to finally answer the question: `why do target networks stabilise TD learning'? To do so, we formalise the notion of a partially fitted policy evaluation method, which describes the use of target networks and bridges the gap between fitted methods and semigradient temporal difference algorithms. Using this framework we are able to uniquely characterise the so-called deadly triad - the use of TD updates with (nonlinear) function approximation and off-policy data - which often leads to nonconvergent algorithms. This insight leads us to conclude that the use of target networks can mitigate the effects of poor conditioning in the Jacobian of the TD update. Instead, we show that under mild regularity conditions and a well tuned target network update frequency, convergence can be guaranteed even in the extremely challenging off-policy sampling and nonlinear function approximation setting.

Tables

Table 1. Table 1 : Relevant Parameters for Cartpole Experiment

Parameter	Value
Environment Parameters
$\gamma$	0.99
Architecture Parameters
MLP Hidden Layers	2
Hidden Layer Size	32
Nonlinearity	ReLU
$ϵ$	0.05
Training Parameters
Total Target Network Updates	500
Learning Rate	[0.001, 0.0005]
Momentum ($\mu$)	[0, 0.01]
Batch Size	500
Steps per Target Network Update ($k$)	5
Data Gathering Steps per Update	5
Replay Buffer Size	2500

Equations

$${\mathcal{T}}^{\pi }[Q^{\pi }](s,a):={\mathbb{E}}_{s^{\prime },a^{\prime },r\sim P_{sar}^{\pi }(s,a)}\left[r+\gamma Q^{\pi }(s^{\prime },a^{\prime })\right].$$

$${\omega }_{i+1}=$$

$${\omega }_i+{\alpha }_i\left(r+\gamma Q_{{\omega }_i}(s^{\prime },a^{\prime })-Q_{{\omega }_i}(s,a)\right){\nabla }_{\omega }{Q}_{{\omega }_i}(s,a),$$

$$\delta (\omega ,{\omega }^{\prime },\varsigma ):=\left(r+\gamma Q_{{\omega }^{\prime }}(s^{\prime },a^{\prime })-Q_{\omega }(s,a)\right){\nabla }_{\omega }{Q}_{\omega }(s,a),$$

$${\omega }_{i+1}={\omega }_i+{\alpha }_i\delta ({\omega }_i,{\omega }_i,\varsigma ).$$

$${\omega }^{\star }\in {\Omega }^{\star }:=\left\{\omega |\delta (\omega ,\omega )=0\right\}.$$

$${\omega }_{kl+i+1}$$

$${\bar{\omega }}_{l+1}$$

$$\mathcal{L}(\omega ;{\bar{\omega }}_l):=\Vert Q_{\omega }-{\mathcal{T}}^{\pi }[Q_{{\bar{\omega }}_l}]{\Vert }_{d,\mu },$$

$${\bar{\omega }}_{l+1}=g^k({\bar{\omega }}_l,\mathcal{D},{\alpha }_l),\mathcal{D}\sim P_{\mathcal{D}},$$

$$\bar{H}(\omega ,{\omega }^{\star };{\bar{\omega }}_l)$$

$${\bar{J}}_{\delta }(\omega ,{\omega }^{\star };{\bar{\omega }}_l)$$

$${\bar{J}}_{\text{TD}}(\omega ,{\omega }^{\star })$$

$${\bar{\omega }}_l^{\star }-{\omega }^{\star }$$

$$=\prod _{i=0}^{l-1}\left(\bar{H}({\bar{\omega }}_{i+1}^{\star },{\omega }^{\star };{\bar{\omega }}_i^{\star }{)}^{-1}{\bar{J}}_{\delta }({\bar{\omega }}_i^{\star },{\omega }^{\star };{\omega }^{\star })\right)({\bar{\omega }}_0-{\omega }^{\star }).$$

$$\Phi$$

$${\Phi }^{\prime }$$

$${\bar{J}}_{\text{TD}}(\omega ,{\omega }^{\star })=\gamma {\Phi }^{\prime }-\Phi .$$

$$\sup \lambda \left({\bar{J}}_{\text{TD}}(\omega ,{\omega }^{\star })\right)$$

$$\le \underset{\omega }{\sup }\sup \lambda (\mathbb{E}\left[({\mathcal{T}}^{\pi }[Q_{\omega }]-Q_{\omega }){\nabla }_{\omega }^2{Q}_{\omega }\right]$$

$$+\mathbb{E}\left[\left(\gamma {\mathbb{E}}^{\prime }[{\nabla }_{\omega }{Q}_{\omega }^{\prime }]-{\nabla }_{\omega }{Q}_{\omega }\right){\nabla }_{\omega }{Q}_{\omega }^{\top }\right]).$$

$${\omega }^{\top }\mathbb{E}\left[\left(\gamma {\mathbb{E}}^{\prime }[{\nabla }_{\omega }{Q}_{\omega }^{\prime }]-{\nabla }_{\omega }{Q}_{\omega }\right){\nabla }_{\omega }{Q}_{\omega }^{\top }\right]\omega$$

$$\le (\gamma -1){\omega }^{\top }\mathbb{E}\left[{\nabla }_{\omega }{Q}_{\omega }{\nabla }_{\omega }{Q}_{\omega }^{\top }\right]\omega <0,$$

$${\delta }_{\text{Reg}}(\omega ,{\omega }^{\prime })=\delta (\omega ,{\omega }^{\prime })+\rho (\omega ,{\omega }^{\prime }),$$

$${\delta }_{\text{Reg}}(\omega ,{\omega }^{\prime })=$$

$$+(1-\mu )\left(\delta ({\omega }^{\prime },\omega )-\eta (\omega -{\omega }^{\prime })\right),$$

$${\bar{H}}_{\text{Reg}}(\omega ,{\omega }^{\star };{\bar{\omega }}_l)$$

$$-(1-\mu )\left({\bar{J}}_{\delta }(\omega ,{\omega }^{\star };{\bar{\omega }}_l)-I\eta \right),$$

$${\bar{J}}_{\delta ,\text{Reg}}(\omega ,{\omega }^{\star };{\bar{\omega }}_l)$$

$$-(1-\mu )\left(\bar{H}(\omega ,{\omega }^{\star };{\bar{\omega }}_l)-I\eta \right).$$

$$\underset{\omega ,{\omega }^{\prime }\in {\mathcal{X}}_{\text{FPE}}({\omega }^{\star })}{\sup }\Vert {\bar{H}}_{\text{Reg}}({\omega }^{\prime },{\omega }^{\star };\omega {)}^{-1}{\bar{J}}_{\delta ,\text{Reg}}(\omega ,{\omega }^{\star };{\omega }^{\star })\Vert <1.$$

$${\lambda }_H^{\star }$$

$$\Vert {\bar{J}}_{\text{FPE}}^{\star }\Vert$$

$$\Vert {\bar{J}}_{\text{TD}}^{\star }\Vert$$

$$\mathcal{C}({\alpha }_l,k)$$

$$+\left(1+{\left|1-{\alpha }_l{\lambda }_H^{\star }\right|}^{k-1}\right)\Vert {\bar{J}}_{\text{FPE}}^{\star }\Vert .$$

$${\sigma }_k$$

$$\mathbb{E}\left[\Vert {\bar{\omega }}_{l+1}-{\omega }^{\star }\Vert \right]\le \mathcal{C}({\alpha }_l,k)\mathbb{E}\left[\Vert {\bar{\omega }}_l-{\omega }^{\star }\Vert \right]+{\alpha }_l{\sigma }_k.$$

Full text

Machine Learning, ICML

1 Introduction

Since their introduction in deep $Q$-networks (DQN) a decade ago (Mnih et al., 2013, 2015), target networks have become a common feature of state-of-the-art deep reinforcement learning algorithms (Lillicrap et al., 2016; Haarnoja et al., 2017, 2018; Fujimoto et al., 2018). Theoretical analysis of target networks has been limited and there has been no satisfactory explanation for their empirical success in stabilising policy evaluation algorithms. Whilst recent analysis has characterised the convergence properties of policy evaluation using target networks (Lee & He, 2019; Fan et al., 2020; Zhang et al., 2021), existing approaches focus on asymptotic results, and usually make simplifying assumptions that neither hold in practice nor account for the true behaviour of target network-based updates. Our work finds that the use of target networks can guarantee that deep RL algorithms will not diverge, even in regimes where traditional RL algorithms fail. Additionally, we establish the first finite-time performance bounds for target networks and general function approximation—without strong simplifying assumptions. Moreover, we prove our key stability assumption can always be satisfied by augmenting our updates with simple $\ell_{2}$ regularisation that does not change the TD fixed points. In doing so, we finally provide theoretical justification for the empirical success that has been observed in challenging, off-policy tasks.

To achieve this, we analyse the use of infrequently updated target value functions by characterising them as a family of methods that we refer to as partially fitted policy evaluation (PFPE). This variant bridges the gap between fitted policy evaluation (FPE) (Le et al., 2019)—which iteratively fit the Bellman backups onto the class of representable function approximators —and classic temporal difference (TD) algorithms (Sutton, 1988) by limiting the fitting phase to a fixed number of steps, precisely reflecting the periodically updated target network algorithms as used in practice.

To characterise the performance of PFPE, we express our algorithm–which has traditionally been viewed through the lens of two-timescale analysis–using a single update applied only to the target network parameters. We show that the stability of the algorithm is determined by analysing the eigenvalues of the Jacobian of this update. This formulation allows us to characterise both the limiting (asymptotic) and finite-time (non-asymptotic) convergence properties of PFPE. Furthermore, it suggests, counterintuitively, that target networks are actually the object being optimised rather than merely a means to stabilise conventional TD updates. This insight leads us to empirically investigate a novel target parameter update scheme that uses a momentum-style update (Polyak, 1964), setting the stage for future research of practical target-based algorithms.

Our bounds on the finite-time performance of PFPE apply to off-policy, nonlinear and partially fitted methods, which have never been investigated previously. We develop key insights into the usefulness of target networks, which we find do not improve asymptotic performance when decaying step sizes are used. Instead, target networks improve the conditioning of TD and fitted methods when the step size does not tend to zero, as is often implemented in practice. Under non-decaying stepsizes, our Jacobian analysis shows how PFPE reconditions the TD Jacobian allowing us to prove convergence in regimes where classic TD methods are unstable, thereby breaking the so-called deadly triad that has plagued TD methods (Sutton & Barto, 2018). Furthermore, our results do not depend on unwieldy assumptions or modifications of algorithms used in practice, such as projection, bounded state spaces, linear function approximation, or iterate averaging, as is done in previous analysis. In addition to our theoretical results, we experimentally evaluate our bounds on a toy domain, indicating that they are tight under relevant hyperparameter regimes. Taken together, our results lead to novel insight as to how exactly target networks affect optimisation, and when and why they are effective, leading to actionable results that can be used to further future research.

2 Preliminaries

Proofs for all theorems, propositions and corollaries can be found in Appendix B

We denote the set of all probability distributions on a set $\mathcal{X}$ as $\mathcal{P}(\mathcal{X})$. We use $\lVert\cdot\rVert$ to denote the $\ell_{2}$-norm. For a matrix $M$, we denote the set of eigenvalues as $\lambda(M)$ with the set of maximum normed eigenvalues as $\lambda_{\max}(M)\coloneqq\operatorname*{arg\,sup}_{\lambda^{\prime}\in\lambda(M)}\lvert\lambda^{\prime}\rvert$ and $\lambda_{\min}(M)\coloneqq\operatorname*{arg\,inf}_{\lambda^{\prime}\in\lambda(M)}\lvert\lambda^{\prime}\rvert$. The $\ell_{2}$-norm (spectral norm) for matrix $M$ is $\lVert M\rVert=\sqrt{\lambda_{\max}(M^{\top}M)}$. Given a function $f:\mathcal{X}\to\mathbb{R}$ and a distribution $\mu\in\mathcal{P}(\mathcal{X})$, we denote the $L_{2}$-norm as: $\lVert f\rVert_{\mu}\coloneqq\sqrt{\mathbb{E}_{x\sim\mu}\left[f(x)^{2}\right]}$.

2.1 Reinforcement Learning

We consider the infinite horizon discounted RL setting. The agent interacts with an environment, formalised as a Markov Decision Process (MDP): $\mathcal{M}\coloneqq\langle\mathcal{S},\mathcal{A},P,P_{0},R,\gamma\rangle$ with state space $\mathcal{S}$, action space $\mathcal{A}$, transition kernel $P:\mathcal{S}\times\mathcal{A}\to\mathcal{P}(\mathcal{S})$, initial state distribution $P_{0}\in\mathcal{P}(\mathcal{S})$, bounded stochastic reward kernel $R:\mathcal{S}\times\mathcal{A}\to\mathcal{P}([-r_{max},r_{max}])$ where $r_{max}\in\mathbb{R}<\infty$ and scalar discount factor $\gamma\in[0,1)$. An agent in state $s\in\mathcal{S}$ taking action $a\in\mathcal{A}$ observes a reward $r\sim R(s,a)$. The agent’s behaviour is determined by a policy that maps a state to a distribution over actions: $\pi:\mathcal{S}\to\mathcal{P}(\mathcal{A})$ and the agent transitions to a new state $s^{\prime}\sim P(s,a)$. We denote the joint distribution of $s^{\prime},a^{\prime},r$ conditioned on $s,a$ for policy $\pi$ as $P^{\pi}_{sar}(s,a)$. We seek to optimise (in the control case), or estimate (in the policy evaluation case) the expected discounted sum of future rewards starting from a given state $s\in\mathcal{S}$. This quantity is given by the state value function, $V^{\pi}(s)=\mathbb{E}_{a\sim\pi(s)}\left[Q^{\pi}(s,a)\right]$, with $Q^{\pi}:\mathcal{S}\times\mathcal{A}\to[-r_{max}/(1-\gamma),r_{max}/(1-\gamma)]$, the action value function, given recursively through the Bellman equation: $Q^{\pi}(s,a)=\mathcal{T}^{\pi}[Q^{\pi}](s,a)$, where the Bellman operator $\mathcal{T}^{\pi}$ projects functions forwards by one step through the dynamics of the MDP:

[TABLE]

$\mathcal{T}^{\pi}$ is a $\gamma$-contractive mapping and thus has a fixed point, which corresponds to the true value of $\pi$ (Puterman, 2014). When estimating MDP values, we employ a value function approximation $Q_{\omega}:\mathcal{S}\times\mathcal{A}\to\mathbb{R}$ parametrised by $\omega\in\Omega\subseteq\mathbb{R}^{n}$.

Many RL algorithms employ TD learning for policy evaluation, which combines bootstrapping, state samples and sampled rewards to estimate the expectation in the Bellman operator (Sutton, 1988). In their simplest form, TD methods update the function approximation parameters according to:

[TABLE]

where $s\sim d,a\sim\mu(s),s^{\prime},a^{\prime},r\sim P^{\pi}_{sar}(s,a)$, $d\in\mathcal{P}(\mathcal{S})$ is a sampling distribution, and $\mu$ is a sampling policy that may be different from the target policy $\pi$. For simplicity of notation and to accommodate the introduction of target networks in Section 3, we define the tuple $\varsigma\coloneqq(s,a,r,s^{\prime},a^{\prime})$ with distribution $P_{\varsigma}$ and the TD-error vector as:

[TABLE]

allowing us to write the TD parameter update as:

[TABLE]

We make the following i.i.d. assumption for clarity of exposition, but discuss other sampling regimes in Appendix D:

Assumption 1**.**

Each $s\sim d$ is drawn i.i.d..

Typically, $d$ is the steady-state distribution of an ergodic Markov chain. We denote the expected TD-error vector as: $\delta(\omega,\omega^{\prime})\coloneqq\mathbb{E}_{\varsigma\sim P_{\varsigma}}[\delta(\omega,\omega^{\prime},\varsigma)]$ and define the set of TD fixed points as:

[TABLE]

If a TD algorithm converges, it converges to a TD fixed point. Convergence of TD methods can only be guaranteed for linear function approximators when sampling on-policy in an ergodic MDP, that is the agent sampling and target distributions are the same. We investigate the phenomenon further as part of our asymptotic analysis in Section 4.1.

3 Partially Fitted Policy Evaluation

Unfortunately, real-world applications of RL often demand the expressiveness of nonlinear function approximators like neural networks and/or the ability to use data that has been collected off-policy, i.e., by following a policy $\mu$ that differs from the target policy $\pi$ for policy evaluation.

3.1 Fitted v Partially Fitted Policy Evaluation

Fitted methods improve on the sample efficiency and stability of TD methods by explicitly incorporating the limitations of the function approximation class through the use of a projection operator (Tsitsiklis & Van Roy, 1997). These methods generally perform some variant of the iterate $Q_{\bar{\omega}_{l+1}}=\Pi^{d^{\pi}}\mathcal{T}^{\pi}Q_{\bar{\omega}_{l}}$ where $\Pi^{d}$ is the projection operator $\Pi^{d}Q=\operatorname*{arg\,min}_{Q^{\prime}}\lVert Q^{\prime}-Q\rVert_{d,\mu}$. These updates are known as fitted policy evaluation (PFE).

The projection step is needed to accommodate the fact that values generally cannot be exactly represented with function approximation. To obtain a practical way of carrying out the PFE updates, a separate set of target parameters can be introduced $\bar{\omega}_{l}\in\Omega$ that parameterise the TD target and are updated every $k$ timesteps:

[TABLE]

The function approximator update in Equation 7 carries out $k$ iterations of stochastic gradient descent (SGD) on the loss:

[TABLE]

before updating the target parameters. In the limit as $k\rightarrow\infty$, assuming convergence of SGD to a global minimum, fully fitted policy evaluation occurs by finding $\omega_{\infty}\in\operatorname*{arg\,inf}_{\omega\in\Omega}\mathcal{L}(\omega,\bar{\omega}_{l})$.

In practice $k$ is finite and only partial policy evaluation occurs before updating the target parameters, a setting we call partially fitted policy evaluation (PFPE). Without loss of generality, we assume that $\bar{\omega}_{0}$ is deterministic with $\lVert\bar{\omega}_{0}\rVert<\infty$ and $\alpha_{i}=\alpha_{l}$ for all $kl\leq i<k(l+1)$, that is stepsizes only change after updating target parameters. As the target parameters are updated to the approximator parameters every $k$ timesteps in Equation 8, it suffices to consider the target parameter update in isolation when analysing PFPE. Our goal is thus to analyse a single update for the target parameters in the canonical form:

[TABLE]

where $\mathcal{D}\coloneqq\{\varsigma_{i}\}_{i=1}^{k}$ is a set of $k$ samples from the environment with distribution $P_{\mathcal{D}}$ and $g^{k}(\bar{\omega}_{l},\mathcal{D}_{l},\alpha_{l})$ reduces the $k$ nested updates from Equation 7 into a single update for the target parameters.

3.2 Jacobian Analysis

In our analysis, we show that the stability of the expected PFPE update $g^{k}(\bar{\omega}_{l},\alpha_{l})\coloneqq\mathbb{E}_{\mathcal{D}\sim P_{\mathcal{D}}}\left[g^{k}(\bar{\omega}_{l},\mathcal{D},\alpha_{l})\right]$ is determined by the conditioning of three Jacobians. We denote the Hessian of the loss as: $H(\omega;\bar{\omega}_{l})\coloneqq\nabla^{2}_{\omega}\mathcal{L}(\omega;\bar{\omega}_{l})$, the Jacobian of the TD-error vector as: $J_{\delta}(\omega;\bar{\omega}_{l})\coloneqq\nabla_{\omega^{\prime}}\delta(\omega,\omega^{\prime})|_{\omega^{\prime}=\bar{\omega}_{l}}$ and define the TD Jacobian as: $J_{\textrm{TD}}(\bar{\omega}_{l})\coloneqq\nabla_{\omega}\delta(\omega,\omega)|_{\omega=\bar{\omega}_{l}}$. Observe that $J_{\textrm{TD}}(\bar{\omega}_{l})=J_{\delta}(\bar{\omega}_{l},\bar{\omega}_{l})-H(\bar{\omega}_{l};\bar{\omega}_{l})$. Without loss of generality, we assume that the Hessian matrix is diagonalisable because, if it is not, an arbitrarily small perturbation can make its eigenvalues distinct and therefore diagonalisable. So that these matrices exist, we require that the expected PFPE update is differentiable almost everywhere, a condition that is guaranteed by a Lipschitz assumption. We also require that the variance of the updates is bounded, motivating the following regularity assumption:

Assumption 2** (Function Approximator Regularity).**

We assume that $\delta(\omega,\omega^{\prime},\varsigma)$ is Lipschitz in $\omega,\omega^{\prime}$ with constant $L$: $\lVert\delta(\omega_{1},\omega^{\prime}_{1},\varsigma)-\delta(\omega_{2},\omega^{\prime}_{2},\varsigma)\rVert\leq L(\lVert\omega_{1}-\omega_{2}\rVert+\lVert\omega_{1}^{\prime}-\omega_{2}^{\prime}\rVert)$ and $\Omega$ is convex, $\mathbb{V}_{\varsigma\sim P_{\varsigma}}[\delta(\omega,\omega,\varsigma)]\coloneqq\mathbb{E}_{\varsigma\sim P_{\varsigma}}[\lvert\delta(\omega,\omega,\varsigma)-\delta(\omega,\omega)\rVert^{2}]\leq\sigma^{2}_{\delta}$ for some $\sigma_{\delta}^{2}<\infty$.

The bounded variance assumption can easily be achieved for unbounded function approximators by truncating the TD error vector, much like the commonly used gradient clipping in gradient descent. We now introduce the path-mean Jacobians, which are the principal element of our analysis:

[TABLE]

Intuitively, a path-mean Jacobian is the average of all of the Jacobians along the line joining $\omega$ to $\omega^{\star}$. The convexity assumption in 2 ensures that the line integral joining any two points in $\Omega$ always exists. The Lipschitz assumption in 2 is only required for Section 4 and can be weakened to any condition that ensures the path-mean Jacobians exist for the remainder of the paper.

Our analysis in Section 4 proves that stability of TD and PFPE under decaying stepsizes is determined solely by the negative definiteness of the TD path-mean Jacobian $\bar{J}_{\textrm{TD}}(\omega,\omega^{\star})$. In Section 5, we show for a non-diminishing stepsize regime that through suitable regularisation (which does not affect the TD fixed point), PFPE’s stability can be determined only by $\alpha_{l}$ and $k$, for which stable values exists. As $\bar{H}(\omega,\omega^{\star};\bar{\omega}_{l})$ is the path-mean Hessian of the loss, convergence can be guaranteed under the same mild assumptions required to prove convergence of a stochastic gradient descent algorithm to minimise $\mathcal{L}(\omega;\bar{\omega}_{l})$. This implies that PFPE can converge under regimes where TD will not as $\bar{J}_{\textrm{TD}}(\omega,\omega^{\star})$ is positive definite.

3.3 Analysis of PFE

We now showcase the power of our Jacobian analysis by writing the PFE updates exactly in terms of $(\bar{\omega}_{0}-\omega^{\star})$:

Theorem 1**.**

Under 2, the sequence of PFE updates $\bar{\omega}_{l+1}^{\star}\in\operatorname*{arg\,inf}_{\omega}\mathcal{L}(\omega,\bar{\omega}_{l}^{\star})$ satisfy:

[TABLE]

We can use Theorem 1 to determine the stability of FPE updates. If $\sup_{\omega,\omega^{\prime}\in\Omega}\left\lVert\bar{H}(\omega^{\prime},\omega^{\star};\omega)^{-1}\bar{J}_{\delta}(\omega,\omega^{\star};\omega^{\star})\right\rVert<1$ then the FPE updates are a contraction mapping and will converge to a fixed point under the Banach fixed-point theorem. We discuss the convergence of FPE under varying regularisation schemes in Section 5.1.

4 Asymptotic Analysis

We now study the behaviour of Equation 10 in the limit of $l\rightarrow\infty$. We introduce the standard Robbins-Munro condition for the decaying stepsizes that is a necessary condition to ensure convergence to a fixed point:

Assumption 3** (Robbins-Munro).**

Each $\alpha_{l}$ is a positive scalar with $\sum_{l=0}^{\infty}\alpha_{l}=\infty$ and $\sum_{l=0}^{\infty}\alpha_{l}^{2}<\infty$.

Now we introduce a core necessary assumption to prove stability of PFPE with diminishing stepsizes:

Assumption 4** (TD Stability).**

There exists a region $\mathcal{X}_{\textrm{TD}}(\omega^{\star})$ containing a fixed point $\omega^{\star}$ such that $\bar{J}_{\textrm{TD}}(\omega,\omega^{\star})$ has strictly negative eigenvalues for all $\omega\in\mathcal{X}_{\textrm{TD}}(\omega^{\star})$.

The key insight from 4 is that the stability of PFPE under diminishing stepsizes is determined only by the eigenvalues of the single step path-mean Jacobian $\bar{J}_{\textrm{TD}}(\omega,\omega^{\star})$, regardless of the value of $k$ or $\alpha_{l}$. Indeed, stochastic approximation can be shown to be provably divergent if this condition cannot be satisfied (Pemantle, 1990). From this perspective, if TD diverges then so will PFPE under diminishing stepsizes, hence the asymptotic stability of PFPE is independent of $k$ and $\alpha_{l}$, and, unlike updating under a two-timescale regime, introducing target parameters that are updated periodically every $k$ timesteps does not improve asymptotic convergence properties under this analysis. Once 4 has been established, there are several approaches to prove convergence of the PFPE update under varying sampling conditions and projection assumptions. We follow the proof of (Vidyasagar, 2022), but discuss approaches that generalise our assumptions in Appendix D

Theorem 2**.**

Let Assumptions 1 to 4 hold. If there exists some fixed point $\omega^{\star}$ with region of contraction $\mathcal{X}_{\textrm{TD}}(\omega^{\star})$ and timestep $t$ such that $\bar{\omega}_{l}\in\mathcal{X}_{\textrm{TD}}(\omega^{\star})$ for all $l\geq t$ the the sequence of target parameter updates in Equation 8 converge almost surely to $\omega^{\star}$.

4.1 The Deadly Triad

We have established that it is not possible to prove convergence of PFPE under diminishing stepsizes if 4 does not hold. We now discuss how adherence to 4 formalises a phenomenon known as the deadly triad (Sutton & Barto, 2018) where it has been established that TD cannot be proved to converge when using function approximators in the off-policy setting. To control for the effect of nonlinear function approximation, we first investigate linear function approximators of the form $Q_{\omega}(s,a)=\phi(s,a)^{\top}\omega$ where $\phi:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}^{n}$ is a feature vector. Define the one-step lookahead distribution as: $P^{\mu}\coloneqq\mathbb{E}_{s\sim d,a\sim\mu(s)}\left[P(s,a)\right]$. Introducing the shorthand:

[TABLE]

we can derive the TD Jacobian as:

[TABLE]

We now examine why the conditioning of $\bar{J}_{\textrm{TD}}(\omega,\omega^{\star})$ explains this phenomenon.

Linear Function Approximation

For linear function approximators, we show in Section A.1 that $\gamma\lVert Q_{\omega}\rVert_{P^{\mu},\pi}<\lVert Q_{\omega}\rVert_{d,\mu}$ for all $\omega$ is a sufficient condition for $\gamma\Phi^{\prime}-\Phi$ to have negative eigenvalues, thereby satisfying 4. This implies that the function approximator class remains non-expansive under the one-step lookahead distribution $P^{\mu}$, thereby preventing the function approximator diverging as the Markov chain is traversed. This condition has been introduced previously in the fitted $Q$-iteration literature (Wang et al., 2020, 2021) as a “low distribution shift” assumption.

In the on-policy setting in an ergodic MDP, we can prove that there exists a stationary distribution $d^{\pi}$ induced by following the target policy $\pi$, that is $\mu=\pi$. Moreover it is assumed that samples come from $d^{\pi}$; hence by the definition of ergodicity, the one-step lookahead distribution is the stationary distribution: $P^{\pi}=d^{\pi}$. It thus follows that $\gamma\lVert Q_{\omega}\rVert_{P^{\mu},\pi}=\gamma\lVert Q_{\omega}\rVert_{d^{\pi},\pi}<\lVert Q_{\omega}\rVert_{d^{\pi},\pi}$ and hence 4 holds automatically for on-policy TD in an ergodic MDP, thereby establishing the convergence properties as a special case via Theorem 2.

For off-policy data, it is not possible to prove that $\gamma\lVert Q_{\omega}\rVert_{P^{\mu},\pi}<\lVert Q_{\omega}\rVert_{d,\mu}$ holds without further assumptions on the sampling policy and MDP. In general, it is not possible to show that $\bar{J}_{\textrm{TD}}(\omega,\omega^{\star})$ is negative definite in the off-policy case as the distribution shift may be too high: there exist counterexample MDPs where off-policy algorithms such as $Q$-learning provably diverge under linear function approximation (Williams & Baird, 1993; Baird, 1995a).

Nonlinear Function Approximation

Even in an on-policy regime, we cannot prove convergence of TD when nonlinear function approximators such as neural networks are used. In these cases, the path-mean Jacobian may not have a closed form solution. However, it can be bounded by the following norm (see Section A.2):

[TABLE]

Even making the same assumption as in Section 4.1 of sampling on-policy in an ergodic MDP to show that

[TABLE]

we cannot prove the negative definiteness of $\bar{J}_{\textrm{TD}}(\omega,\omega^{\star})$ required to satisfy 4. This is because the matrix $\mathbb{E}\left[(\mathcal{T}^{\pi}[Q_{\omega}]-Q_{\omega})\nabla^{2}_{\omega}Q_{\omega}\right]$ can be arbitrarily positive definite depending on the MDP and choice of function approximator. Indeed, there exist counterexample MDPs with provably divergent nonlinear function approximators when sampling on-policy (Tsitsiklis & Van Roy, 1997).

5 Non-asymptotic Analysis

Our asymptotic analysis in Section 4 shows that increasing $k$ or adjusting $\alpha_{l}$ for PFPE does not affect the asymptotic strong convergence properties of the TD algorithm, implying that target networks do not stabilise TD if stepsizes tend to zero. We showed that the underlying reason for this was the deadly triad, which we formalised as adherence to 4. We now replace 4, that is $\bar{J}_{\textrm{TD}}(\omega,\omega^{\star})$ is negative definite, with the assumption that FPE is stable:

Assumption 5** (FPE Stability).**

There exists a region $\mathcal{X}_{\textrm{FPE}}(\omega^{\star})$ containing a fixed point $\omega^{\star}$ such that $\sup_{\omega,\omega^{\prime}\in\mathcal{X}_{\textrm{FPE}}(\omega^{\star})}\left\lVert\bar{H}(\omega^{\prime},\omega^{\star};\omega)^{-1}\bar{J}_{\delta}(\omega,\omega^{\star};\omega^{\star})\right\rVert<1$.

5.1 Stabilising FPE

We now prove that 5 can always be satisfied using regularisation schemes that do not affect the TD fixed points. We introduce the following regularised TD vector:

[TABLE]

where $\rho(\omega,\omega^{\prime})$ is a regularisation term such that $\rho(\omega,\omega)=0$, thereby not changing the TD fixed point or TD update. As an example, $\rho(\omega^{\prime},\omega^{\prime})$ can contain powers of regularisation terms $M_{\textrm{Reg}}(\omega-\omega^{\prime})$ in addition to combinations of $\delta(\omega^{\prime},\omega)$ and $\delta(\omega,\omega^{\prime})$ terms, where $\delta(\omega^{\prime},\omega)$ is a TD vector with target and $Q$-network parameters swapped. In this paper, we briefly study regularisation of the form:

[TABLE]

where $\mu$ mixes the TD updates and $\eta$ controls the degree of regularisation. We emphasise that $\delta_{\textrm{Reg}}(\bar{\omega}_{l},\bar{\omega}_{l})=\delta(\bar{\omega}_{l},\bar{\omega}_{l})$, leaving the TD update unchanged. In contrast, unless $\omega^{\star}$ is known a priori, introducing regularisation that modifies the TD update—as is done in (Zhang et al., 2021)—will affect the TD fixed points. We now prove that FPE can be stabilised by treating $\eta$ and $\mu$ as hyperparameters to be tuned to the specific MDP.

Proposition 1**.**

Using the regularised TD vector in Equation 26, the path-mean Jacobians are:

[TABLE]

5* is satisfied if:*

[TABLE]

There exists finite $\eta,\mu$ such that Equation 31 holds.

The key insight from Proposition 1 is that regularisation stabilises FPE (and hence PFPE) without affecting existing TD fixed points, even when TD is unstable, motivating future research directions to develop sophisticated regularisation techniques.

5.2 Convergence Analysis

By carrying out a non-asymptotic analysis, we now investigate how the deadly triad can be broken by PFPE using Equation 24 when stepsizes do not tend to zero. This leads to a formal understanding of how target parameters stabilise TD under stepsize regimes that are actually used in practice when classic TD methods fail. The foundation of our analysis is a condition function that can be used to determine the stability of the updates:

Definition 1** (Condition Function).**

For a subset $\mathcal{X}(\omega^{\star})\subseteq\Omega$ with corresponding fixed point $\omega^{\star}\in\mathcal{X}(\omega^{\star})$ such that $\omega_{i}\in\mathcal{X}(\omega^{\star})$ for all $i\geq 0$ , let

[TABLE]

and define the condition function as:

[TABLE]

The condition function depends on the maximal eigenvectors of the Jacobians introduced in Section 3.2, and so can still be used to analyse general nonlinear function approximators for which the path-mean Jacobians have no analytic solution. Using the condition function, we decompose the error at a given timestep into the effect of the expected update plus the error induced by variance of the update:

Theorem 3**.**

Define

[TABLE]

Let Assumptions 1 and 2 hold, then:

[TABLE]

The effect of the expected update (the first term in Equation 38) is bounded by the condition function, which depends both on data conditioning but critically, on both $k$ and $\alpha_{l}$ as well and must diminish with increasing $l$ to ensure convergence. Using this decomposition, we see convergence is guaranteed if the following assumption holds:

Assumption 6** (Contraction Region).**

We assume that $\mathcal{C}(\alpha,k)\leq c<1$ over $\mathcal{X}_{\textrm{FPE}}(\omega^{\star})$.

allowing us to prove convergence of PFPE for stepsizes that don’t tend to zero provided that updates remain in a region of contraction:

Corollary 3.1**.**

Let Assumptions 1, 2, 5 and 6 hold. For a fixed stepsize $\alpha_{l}=\alpha>0$,

[TABLE]

Corollary 3.1 is a key result of this work. Our result demonstrates geometric decay of errors in $l$, to a ball of fixed radius $\frac{\alpha\sigma_{k}}{1-c}$. This is analogous to related work in stochastic gradient descent (Bottou et al., 2018), and matches the intuition that, without decaying stepsize, variance in the updates means that convergence to a fixed point does not occur. Note that the radius of the ball which we converge to can be made arbitrarily small by decreasing $\alpha$.

This supports the use of a hybrid approach, wherein a fixed step size is used until iterates are no longer improving and then reducing step size and repeating to decrease the radius of the ball of convergence whilst maintaining $k$ as small as possible. In the remainder of this section, we explore the properties of the condition function to ensure the existence of a region of contraction satisfying 6.

5.3 Properties of PFPE Condition Function

We now investigate key properties of Equation 36 to understand how target parameters can lead to convergence when classic TD methods fail. If $\bar{J}_{\textrm{TD}}(\omega,\omega^{\star})$ is positive definite, TD is provably divergent, however our analysis reveals that there are values of $k$ and $\alpha_{l}$ for which PFPE does converge.

Property 1: Lower bound

$\left\lVert\bar{J}_{\textrm{FPE}}^{\star}\right\rVert\leq\mathcal{C}(\alpha_{l},k)$.

We first investigate the conditions for which our choice of function approximators can never be used to prove convergence. Our condition function implies that we cannot prove convergence for any $\lambda_{H}^{\star}\leq 0$ or $\lambda_{H}^{\star}\geq\frac{2}{\alpha_{l}}$ as repeated applications of $\lvert 1-\alpha_{l}\lambda_{H}^{\star}\rvert^{2}$ do not reduce the effect the ill-conditioning of $\bar{J}_{\textrm{TD}}(\omega,\omega^{\star})$. We formalise this in the following regularity assumption:

Assumption 7** (Eigenvalue Regularity Assumption).**

Given a region $\mathcal{X}\subseteq\Omega$, for all $\omega,\omega^{\prime}\in\mathcal{X}$ there exists $0<\lambda_{1}^{\min}$ and $\lambda_{1}^{\max}<\infty$ such that $\lambda^{\min}\leq\lambda(\nabla_{\omega}^{2}\mathcal{L}(\omega;\omega^{\prime}))\leq\lambda^{\max}$.

We now propose two simple fixes to avoid this issue. Recall from Section 3.2 that $\lambda_{H}^{\star}$ is an eigenvalue of the Hessian of a loss. If $\lambda_{H}^{\star}$ was negative, this would imply that the Hessian is not positive semidefinite for all $\omega$ in the region of interest; hence we cannot prove convergence of stochastic gradient descent on the loss $\mathcal{L}(\omega;\bar{\omega}_{l})$, let alone the full PFPE algorithm. To remedy this problem, the eigenvalues of the matrix can be increased using the regularisation introduced in Equation 24 without affecting the TD fixed point. However, if $\lambda_{H}^{\star}\geq\frac{2}{\alpha_{l}}$, then the conditioning of the Hessian matrix is ill-suited to the chosen step-size, and an easy remedy is to decrease $\alpha_{l}$. Our bound shows that the condition function is lower bounded by $\lVert J_{\textrm{FPE}}^{\star}\rVert$, and so if 5 does not hold, then convergence of PFPE is not provable.

Property 2: Monotonicity

For $|1-\alpha_{l}\lambda_{H}^{\star}\rvert<1$, $\mathcal{C}(\alpha_{l},k)\leq\mathcal{C}(\alpha_{l},k^{\prime})$ for $k\leq k^{\prime}$.

The monotonicity property ensures that $|1-\alpha_{l}\lambda_{H}^{\star}\rvert<1$ defines the interval of Hessian eigenvalues for which there is a regime in which we can increase $k$ in order to ensure PFPE updates are a contraction mapping. This suggests that a key role of the target network is to help mitigate the effects of the ill-conditioning of the TD Jacobian when using fixed step sizes. We now investigate how decreasing stepsizes and increasing the number of PFPE steps affect the conditioning of PFPE, which validates this hypothesis.

Property 3: Limits

For any $k<\infty$, $\lim_{\alpha_{l}\rightarrow 0}\mathcal{C}(\alpha_{l},k)=\left\lVert\bar{J}_{\textrm{TD}}^{\star}\right\rVert+2\left\lVert\bar{J}_{\textrm{FPE}}^{\star}\right\rVert$. For any $0<\alpha_{l}<\frac{2}{\lambda_{H}^{\star}}$, $\lim_{k\rightarrow\infty}\mathcal{C}(\alpha_{l},k)=\left\lVert\bar{J}_{\textrm{FPE}}^{\star}\right\rVert$.

The first limit illustrates the effects of a diminishing stepsize sequence, confirming our bound is consistent with the results of the previous section that increasing $k$ does not improve the convergence properties of PFPE if stepsizes tend to zero and PFPE only stabilises TD for $0<\alpha_{l}$. By taking the limit $k\rightarrow\infty$, we compliment our monotonicity result, obtaining a bound for how much we can improve on the stability of TD by increasing $k$. As expected, in the limit of $k\rightarrow\infty$, the condition function tends to $\lVert J_{\textrm{FPE}}^{\star}\rVert$. Through this insight, we interpret PFPE as mixing FPE and TD updates according the coefficient $\lvert 1-\alpha_{l}\lambda^{\star}_{H}\rvert^{k-1}$: for $k=1$, PFPE uses only TD updates and in the limit $k\rightarrow\infty$, PFPE recovers the FPE update.

5.4 Breaking the Deadly Triad

We now combine all properties presented in this section into our main result, proving that through suitable regularisation and choice of $\alpha_{l}$ and $k$, PFPE breaks TD’s deadly triad described in Section 4.1:

Theorem 4**.**

Let 7 hold over $\mathcal{X}_{\textrm{FPE}}(\omega^{\star})$ from Definition 1. For any $\frac{1}{\alpha_{l}}>\frac{\lambda_{1}^{\min}+\lambda_{1}^{\max}}{2}$ such that $\alpha_{l}>0$, any

[TABLE]

ensures that $\mathcal{X}_{\textrm{FPE}}(\omega^{\star})$ is a region of contraction satisfying 6.

Theorem 4 demonstrates that appropriate values of $\alpha_{l}$ and $k$ can be found by treating them as hyperparameters, decreasing $\alpha_{l}$ and increasing $k$ until the algorithm is stable, reducing the conditions needed to prove convergence of PFPE to those of proving convergence of stochastic gradient descent on the loss $\mathcal{L}(\omega;\bar{\omega}_{l})$. The key insight of Theorem 4 is that even when TD is unstable due to $1<\lVert I+\alpha_{l}\bar{J}_{\textrm{TD}}(\bar{\omega}_{l},\omega^{\star})\rVert$, there exists a finite $k$ such that $\mathcal{C}(\alpha_{l},k)<1$ and hence PFPE is stable. We illustrate this phenomenon with a sketch in Figure 1, demonstrating that increasing $k$ ensures PFPE is provably convergent in regimes where TD cannot be proved to converge.

The key insight of our analysis is that, unlike in TD where stability can only be proved if the matrix $\bar{J}_{\delta}(\omega,\omega^{\star};\omega^{\star})-\bar{H}(\omega,\omega^{\star};\omega)$ is negative definite, with suitable regularisation, the stability of PFPE can be determined solely by tuning $\alpha_{l}$ and $k$, regardless of the MDP, sampling regime, or function approximator, thereby breaking the deadly triad. The choice of $\alpha_{l}$ and $k$ thus becomes a trade-off between maintaining a fast rate of convergence and reducing the residual variance $(\alpha_{l}\sigma_{k})^{2}$ in Equation 38.

6 Related Work

Our work furthers the analysis of TD, FPE, and target-network based methods. In this section we provide a brief overview of previous investigations of these algorithms.

Fitted Policy Evaluation

FPE is a relatively well understood class of RL algorithms from a theoretical perspective. Nedić & Bertsekas (2003) analyse the convergence of the Least-Squares Policy Evaluation (LSPE) of Bertsekas & Ioffe (1996) in an on-policy, linear function approximation setting. Analysis of LSPE shows that learning with constant step size leads to theoretical and empirical gains compared to TD and LSPE with decaying step sizes (Bertsekas et al., 2004), which mirrors our conclusions in Section 5.4.

In the context of fitted methods applied to off-policy and control problems, Munos & Szepesvári (2008) prove generalisation properties of Fitted $Q$ Iteration (Ernst et al., 2005) for general function classes under assumptions of low projection error and limited data distribution shift. Le et al. (2019) coin the term FPE, and formalise the algorithm for general function approximators, with theoretical results under similar assumptions to Munos & Szepesvári (2008).

Theory of TD

Previous results concerning convergence rates of classic TD methods largely argue that the Bellman operator is a contraction, and thus most focus on linear function approximation. Tsitsiklis & Van Roy (1997) first proved convergence of linear, on-policy TD, arguing that the projected Bellman operator in this setting is a contraction. This corresponds to a special case of 4. Dalal et al. (2017) give the first finite time bounds for linear TD(0), under an i.i.d. data model similar to the one that we use here. Bhandari et al. (2018) provide bounds for linear TD in both the i.i.d. data setting and a correlated data setting, through analogy with SGD. Srikant & Ying (2019) approach the problem from the perspective of Ordinary Differential Equations (ODE) analysis, bounding the divergence of a Lyapunov function from the limiting point of the ODE that arises from the TD update scheme.

Analysis of Target Networks

Existing analysis of the theoretical properties of target networks are limited, usually involving algorithmic changes or restrictive assumptions. Yang et al. (2019) show convergence of a $Q$-learning approach using a target network that is updated using Polyak averaging with nonlinear function approximation. However their analysis–which makes use of two-timescale analysis–requires a projection step to limit the magnitude of parameters. Carvalho et al. (2020) show convergence of a related method using two-timescale analysis, though their target network update differs significantly from those used in practice. Zhang et al. (2021) analyse the use of target networks with linear function approximation, but require projection steps on both the target network and value parameters. Lee & He (2019) provide finite-iteration bounds, but are limited to on-policy data, linear function approximation, and near-perfect fitting to the target network between updates. Fan et al. (2020) analyse the use of target networks for deep Q-learning (Mnih et al., 2015) with the simplifying assumption that they are performing some form of Fitted $Q$ Iteration.

None of these efforts yield finite time bounds with target networks, nor do any match the policy evaluation methods used in practice as well as the PFPE analysis studied here. Furthermore, our use of a single target network update, rather than independent target and value updates leads to simpler bounds without the need for a two-timescale analysis.

GTD and TDC Methods

While not directly related to PFPE or the use of target networks, GTD-style approaches (Sutton et al., 2008, 2009; Maei et al., 2009) also lead to convergent, TD-style algorithms, even with off-policy sampling or nonlinear function approximation. These methods maintain a second set of parameters which must be optimised at a faster timescale than the value parameters. However, these approaches are commonly found to be ineffective and not used in practice due to the difficulty in tuning the rate of second timescale (see, e.g. Fellows et al. (2021)), and potentially additional variance introduced by the second set of parameters (Ghiassian et al., 2020).

Improving Conditioning of TD Methods

Previous work concerning conditioning of TD methods has been largely concerned with approximation of preconditioning approaches to iterative-methods (Saad, 2003). The first such approach was focused on preconditioning of on-policy, linear, least-squares forms of TD (Yao & Liu, 2008). Chen et al. (2020); Romoff et al. (2020) adapt this approach for nonlinear function approximation, though their results are still on-policy. Our work, on the other hand, demonstrates that use of the target network, alongside fixed step sizes, changes the form of parameter iterates to ameliorate the poor conditioning that occurs when directly applying TD or fitted methods, even in off-policy settings.

7 Experiments

We proceed to empirical investigation of our bounds. First, we demonstrate that the use of an infrequently updated target network leads to convergence of off-policy evaluation on the Baird’s notorious counterexample. Then, we evaluate the effect of a speculative modified update rule in the Cartpole-v0 “gym” environment (Brockman et al., 2016). Additional implementation details for both experiments can be found in Appendix C.

7.1 Baird’s Counterexample

In this experiment, we demonstrate the practicality of our core claim–that for sufficiently high $k$ and low enough $\alpha$, PFPE will not diverge, even under conditions that TD does. To do so, we evaluate the use of target networks with varying update frequencies on the well known off-policy counterexample due to Baird (1995b).

In this environment, depicted in Appendix C, rewards are zero everywhere, transitions are deterministic, and the true solution lies within the linear function approximation class that we make use of. The behaviour policy is set such that all states are sampled with uniform probability. The target policy, however, always transitions to a specific state, and remains there. Due to undersampling of this absorbing state, conventional TD policy evaluation diverges, demonstrating that even in simple environments, TD can be unstable when applied off policy with function approximation.

We report the stepwise (fitted) error in Figure 2 across different values of $k$, for fixed step size $\alpha=0.01$, and fixed discount factor $\gamma=0.99$. We see that with $k=1$–which is equivalent to using TD with fixed step sizes–our parameters diverge. Likewise, if $k$ is set to 5 or 10, we are unable to overcome the conditioning of the TD Jacobian and diverge, albeit at a slower rate. Once we take $k\geq 500$, however, conditioning has improved enough to lead to convergence. This supports our theoretical conclusion: that PFPE can be used to improve the convergence conditions of TD.

7.2 Cartpole Experiment

One important insight of our analysis is that we can view the entire optimisation process as a sequence of updates to the target network only. This suggests investigation into alternative forms or acceleration of target network updates. Inspired by the use of optimisation methods with momentum in RL settings (Sarigül & Avci, 2018; Haarnoja et al., 2018), we investigate the effects of a target network that is updated using momentum.

Unlike the standard periodic target network update in Equation 8, we postulate that there may be settings in which a periodic update with momentum may accelerate or stabilise convergence. This update works as follows:

[TABLE]

We investigate the effects of this momentum update on the Cartpole domain. For this experiment, we use control results in which the policy is continuously learned. This is because control problems are inherently off-policy, and induce additional instability, and thus benefit from faster and more stable convergence of values. We implement the standard DQN (Mnih et al., 2015) algorithm, with our modified target network update in order to examine its effect. The results are shown in Figure 3. Our proposed update indeed leads to improved learning and stability, at least for the hyperparameter ranges tested, suggesting that the momentum update has merit. As a result, we propose investigation of more sophisticated target network update schemes as an avenue for future research.

8 Conclusions

This work analysed the use of target networks through the formulation of a novel class of TD updates, which we refer to as PFPE. These updates generalise traditional TD(0) and fitted policy evaluation methods. Our analysis contributes asymptotic and finite time bounds without additional restrictive assumptions or significant changes to the algorithms used in practice. In our main result, we uncovered novel insight as to when and how target networks are useful: provided step-sizes don’t tend to zero and FPE is stable, there always exists a finite number of update steps $k$ and non-zero upper bound over stepsizes such that PFPE can improve conditioning to ensure learning is stable when classic TD methods fail. Our focus on the target network update as the object of concern in terms of optimisation suggests that novel, accelerated methods for updating target networks may help speed up and stabilise learning. Our initial experiments support this notion. Moreover, our analysis reveals that regularisation may be key to determining the stability of PFPE, opening a promising avenue for future research.

Acknowledgements

Mattie Fellows is funded by a generous grant from Waymo. We would like to thank Valentin Thomas for providing a helpful discussion.

Appendix A Derivations

A.1 Derivation of 4 from low distributional shift

Starting from 4 and the definition of negative definiteness, we need to show:

[TABLE]

whenever $\gamma\lVert Q_{\omega}\rVert_{P^{\mu},\pi}<\lVert Q_{\omega}\rVert_{d,\mu}$, for all $\omega$. Investigating the first term by expanding the expectations we see:

[TABLE]

This allows us to apply our assumption:

[TABLE]

A.2 Nonlinear Jacobian Analysis

We start by bounding the maximum eigenvalue:

[TABLE]

We now substitute for the definition of the TD Jacobian, yielding:

[TABLE]

as required.

Appendix B Proofs

B.1 FPE Analysis

Lemma 1**.**

Under 2, the FPE update $\bar{\omega}_{l+1}\in\operatorname*{arg\,inf}_{\omega}\mathcal{L}(\omega,\bar{\omega}_{l})$ satisfies:

[TABLE]

Proof.

Given $\bar{\omega}_{l}$, the FPE fixed point $\bar{\omega}_{l}^{\star}$ must be an element of the set:

[TABLE]

which we use to derive a stability condition for the projection operator:

[TABLE]

Let $\ell_{1}(t)\coloneqq\bar{\omega}_{l}^{\star}-t(\bar{\omega}_{l}^{\star}-\omega^{\star})$ and $\ell_{2}(t)\coloneqq\bar{\omega}_{l}-t(\bar{\omega}_{l}-\omega^{\star})$. We introduce the notation:

[TABLE]

We observe that $\delta_{1}(0,\bar{\omega}_{l})=\delta(\bar{\omega}_{l}^{\star},\bar{\omega}_{l})$ and $\delta_{1}(1,\bar{\omega}_{l})=\delta(\omega^{\star},\bar{\omega}_{l})$, and $\delta_{2}(0,\omega^{\star})=\delta(\omega^{\star},\bar{\omega}_{l})$ and $\delta_{2}(1,\omega^{\star})=\delta(\omega^{\star},\omega^{\star})$. From the fundamental theorem of calculus and 2, it follows:

[TABLE]

as required. ∎

Theorem 1**.**

Under 2, the sequence of FPE updates $\bar{\omega}_{l+1}^{\star}\in\operatorname*{arg\,inf}_{\omega}\mathcal{L}(\omega,\bar{\omega}_{l}^{\star})$ satisfy:

[TABLE]

Proof.

From Equation 61 of Lemma 1, it follows:

[TABLE]

Recursively applying the result $l$ times, our result follows immediately. ∎

B.2 Asymptotic Analysis

For this section, we define a Martingale difference sequence that captures the behaviour of our updates. Let $\{\omega_{i}\}_{i=0}^{k}$ denote the intermediate function approximation parameters between target parameter updates $\bar{\omega}_{l+1}$ and $\bar{\omega}_{l}$, with $\omega_{0}=\bar{\omega}_{l}$ and $\omega_{k}=\bar{\omega}_{l+1}$. We start by writing our target parameter updates as:

[TABLE]

where we define $h_{i}(\bar{\omega}_{l},\mathcal{D},\alpha_{l})$ recursively as:

[TABLE]

and remark that $h_{0}(\bar{\omega}_{l},\mathcal{D},\alpha_{l})=0$ trivially. We write our target parameters updates as:

[TABLE]

where

[TABLE]

and $\mathcal{M}_{l+1}$ defines the Martingale sequence:

[TABLE]

In this section, we demonstrate that the proof of Borkar & Meyn (2000, Theorem 2.2) can be adapted to account for the additional term $\varepsilon_{l+1}$ that arises due to the use of target networks in the updates. Lemma 2 demonstrates that as stepsizes tend to zero, the effect of $\epsilon_{l+1}$ becomes negligible, hence the inclusion of $\varepsilon_{l+1}$ negligible to our analysis of the underlying ODE defined by the TD updates.

Lemma 2**.**

Let $\nu_{n,n+m}\coloneqq\sum_{l=n}^{m+n-1}\alpha_{l}\epsilon_{l+1}$ for $m\geq 1$. Under Assumptions 1 to  3, $\lim_{n\rightarrow\infty}\sup_{m}\lVert\nu_{n,n+m}\rVert=0$ almost surely.

Proof.

We start by bounding each $\lVert\epsilon_{i+1}\rVert$ using the the Lipschitzness of $\delta$ from 2:

[TABLE]

To proceed, we recognise that each $\lVert h_{i}(\bar{\omega}_{l},\mathcal{D}_{l},\alpha_{l})\rVert\leq c_{h}<\infty$ almost surely where $c_{h}$ is a finite positive constant - otherwise:

[TABLE]

for at least one $i>j$, hence $\mathbb{V}_{\varsigma\sim P_{\varsigma}}[\delta(\omega,\omega^{\prime},\varsigma)]=\infty$ for some $\omega,\omega^{\prime}$ thereby violating 2. Using $c_{h}$, we bound $\lVert\epsilon_{l+1}\rVert$:

[TABLE]

almost surely. We use this result to bound $\lVert\nu_{n,n+m}\rVert$:

[TABLE]

Now, under 3,

[TABLE]

hence by the bound established in Equation 95:

[TABLE]

almost surely, as required. ∎

Theorem 2**.**

Under Assumptions 1- 4, the sequence of target parameter updates in Equation 8 converge almost surely to $\omega^{\star}$.

Proof.

Our update

[TABLE]

is identical to the update presented in Borkar & Meyn (2000, Eq. 2.1.1) with an additional term $\varepsilon_{l+1}$. Proof of convergence to the ODE is given by Borkar & Meyn (2000, Lemma 1), which is predicated on the convergence of:

[TABLE]

from Borkar & Meyn (2000, Eq. 2.1.6) where

[TABLE]

for $n\geq 1$, that is $\lim_{n\rightarrow\infty}\sup_{m}\lVert\Delta_{n,n+m}\rVert=0$, almost surely. To adapt our updates so that Borkar & Meyn (2000, Lemma 1) still applies, we recognise that the term $\zeta_{n}$ is now replaced in our updates with:

[TABLE]

and hence $\Delta_{n,n+m}$ is replaced in our updates with:

[TABLE]

where $\nu_{n,n+m}$ is defined as Lemma 2. All arguments of Borkar & Meyn (2000, Lemma 1) remain unchanged, except Eq. 2.1.9, where we must now show that $\lim_{n\rightarrow\infty}\sup_{m}\lVert\bar{\Delta}_{n,n+m}\rVert=0$:

[TABLE]

Applying Lemma 2 yields $\lim_{n\rightarrow\infty}\sup_{m}\lVert\nu_{n,n+m}\rVert=0$ almost surely, hence

[TABLE]

which is proved in Borkar & Meyn (2000, Lemma 1). Convergence of our algorithm is thus only predicated on the convergence of the update:

[TABLE]

Borkar & Meyn (2000, Theorem 2.2) proves convergence of Equation 111 almost surely to $\omega^{\star}$ given the following four conditions hold:

1. I

$k\delta(\omega,\omega)$ is Lipschitz in $\omega$, 2. II

Stepsizes $\alpha_{l}$ satisfy 3, 3. III

The sequence $\{\mathcal{M}_{l},\mathcal{F}_{l}\}_{l\geq 0}$ is a Martingale difference sequence with respect to the increasing family of $\sigma$-algebras: $\mathcal{F}_{l}\coloneqq\sigma(\{\bar{\omega}_{i},\mathcal{M}_{i}\}_{i\in\{0:l\}})$ where $\mathbb{E}\left[\mathcal{M}_{l+1}|\mathcal{F}_{l}\right]=0$ and $\mathbb{E}\left[\lVert\mathcal{M}_{l+1}\rVert^{2}|\mathcal{F}_{l}\right]\leq C(1+\lVert\bar{\omega}_{l}\rVert^{2})$ for some positive $C<\infty$. 4. IV

The sequence of iterates remain bounded, that is $\sup_{l}\lVert\bar{\omega}_{l}\rVert<\infty$ almost surely.

Conditions I and II hold trivially.

For Condition III, we can take expectations of the Martingale difference:

[TABLE]

as required. We now show that the variance is bounded using 2:

[TABLE]

thereby satisfying Condition III.

Finally, we prove Condition IV using Vidyasagar (2022, Theorem 5), which states iterates remain bounded almost surely if:

1. (a)

Conditions I and III hold; 2. (b)

there exists some Lyapunov function $V:\Omega\mapsto\mathbb{R}^{+}$ such that $a\lVert\omega-\omega^{\star}\rVert^{2}\leq V(\omega)\leq b\lVert\omega-\omega^{\star}\rVert^{2}$ for constants $a,b>0$ and $\lVert\nabla^{2}_{\omega}V(\omega)\rVert$ is bounded, and; 3. (c)

$\nabla_{\omega}V(\omega)^{\top}\delta(\omega,\omega)<0$ for all $\omega\in\mathcal{X}_{\textrm{TD}}(\omega^{\star})$.

We propose $V(\omega)=\frac{1}{2}\lVert\omega-\omega^{\star}\rVert^{2}$ as a candidate Lyapunov function, which trivially satisfies (b). We now show (c) holds by applying the fundamental theorem of calculus to $\delta(\omega,\omega)$. Let $\ell(t)\coloneqq\omega-t(\omega-\omega^{\star})$. Like in Theorem 1, it follows:

[TABLE]

hence:

[TABLE]

for all $\omega\in\mathcal{X}_{\textrm{TD}}(\omega^{\star})$ under 4, as required. ∎

B.3 Stabilising FPE

Proposition 1**.**

Using the regularised TD vector in Equation 26, the path-mean Jacobians are:

[TABLE]

5* is satisfied if:*

[TABLE]

There exists finite $\eta,\mu$ such that Equation 130 holds.

Proof.

Taking derivatives of $\delta_{\textrm{Reg}}(\omega,\omega^{\prime})$:

[TABLE]

For clarity, we drop arguments of $\omega^{\prime},\omega^{\star}$ and $\omega$ from our notation.

[TABLE]

We note that $\left(\bar{J}_{\delta}-\bar{H}-I\eta\right)$ can always be made non-singular (and hence invertible) through an arbitrarily small change in $\eta$, allowing us to multiply the first term by $\left(\bar{J}_{\delta}-\bar{H}-I\eta\right)^{-1}\left(\bar{J}_{\delta}-\bar{H}-I\eta\right)=I$, yielding:

[TABLE]

We observe that:

[TABLE]

hence taking limits $\eta\rightarrow\infty$ yields:

[TABLE]

From the continuity of the norm, it follows:

[TABLE]

implying that $\lim_{\eta\rightarrow\infty}\lVert\bar{H}_{\textrm{Reg}}(\omega,\omega^{\star};\bar{\omega}_{l})\rVert<1$ for any $\mu>2$, $\mu\in(0,\frac{2}{3})$, which it suffices assume for hereon out. From the definition of the limit, there exists some finite $\eta^{\prime}$ such that

[TABLE]

for all $\eta>\eta^{\prime}$ for some small $0<\epsilon<1$, and hence

[TABLE]

for all $\eta>\eta^{\prime}$, as required. ∎

B.4 Nonasymptotic Analysis

Lemma 3**.**

Under 2, for $i>0$ the expected updates can be factored as:

[TABLE]

and for $i=0$:

[TABLE]

Proof.

By the definition of the expected update $\omega_{i+1}$:

[TABLE]

Like in Theorem 1, let $\ell(t)\coloneqq\omega_{i}-t(\omega_{i}-\bar{\omega}_{l}^{\star})$ define the line connecting $\omega_{i}$ to $\bar{\omega}_{l}^{\star}$. Using this notation we re-write the expected update as:

[TABLE]

Applying the fundamental theorem of calculus under 2 and the chain rule yields our desired result:

[TABLE]

Our second result follows immediately:

[TABLE]

For our final result:

[TABLE]

By the definition of the expected update:

[TABLE]

Let $\ell(t)\coloneqq\bar{\omega}_{l}-t(\bar{\omega}_{l}-\omega^{\star})$ define the line connecting $\bar{\omega}_{l}$ to $\omega^{\star}$. Using this notation we re-write the expected update as:

[TABLE]

Applying the fundamental theorem of calculus under 2 and the chain rule yields our desired result:

[TABLE]

∎

Lemma 4**.**

Under 2,

[TABLE]

Proof.

We start by bounding the expected norm term using Jensen’s inequality: $\mathbb{E}_{X}[\sqrt{X^{2}}]\leq\sqrt{\mathbb{E}_{X}[X^{2}]}$:

[TABLE]

where we applied the triangle inequality to derive the final line. We bound the variance term by substituting $\omega_{i+1}=\omega_{i}+\alpha_{l}\delta(\omega_{i},\bar{\omega}_{l},\varsigma_{i})$:

[TABLE]

Applying Lemma 3 to the expectation and using the triangle inequality yields our desired result:

[TABLE]

∎

Theorem 3**.**

Define

[TABLE]

Let Assumptions 1 and 2 hold, then:

[TABLE]

Proof.

Let $\{\omega_{i}\}_{i=0}^{k}$ denote the intermediate function approximation parameters between target parameter updates $\bar{\omega}_{l+1}$ and $\bar{\omega}_{l}$, with $\omega_{0}=\bar{\omega}_{l}$ and $\omega_{k}=\bar{\omega}_{l+1}$. We define the set of samples up to $i$ as: $\mathcal{D}_{i}\coloneqq\{\varsigma_{j}\}_{j=0}^{i}$ with distribution $P_{\mathcal{D}_{i}}$, with sample $\varsigma_{j}$ having distribution $P_{\varsigma_{j}}$. Under this notation, we must show:

[TABLE]

Applying Lemma 4 to the inner expectation:

[TABLE]

Applying Equation 178 from Lemma 4 to the inner expectation and applying Lemma 3 yields:

[TABLE]

Recursively applying Equation 195 to Equation 189 $k-1$ times yields:

[TABLE]

Now, applying Equation 178 and Lemma 3 to the expectation:

[TABLE]

Substituting into Equation 195:

[TABLE]

Finally, we apply Theorem 1 to yield our desired result:

[TABLE]

∎

Corollary 3.1**.**

Let Assumptions 1, 2, 5 and 6 hold. For a fixed stepsize $\alpha_{l}=\alpha>0$. For a fixed stepsize $\alpha_{l}=\alpha>0$,

[TABLE]

Proof.

We start by applying Theorem 3:

[TABLE]

As $\mathcal{X}_{\textrm{FPE}}(\omega^{\star})$ is a region of contraction and $\bar{\omega}_{l}\in\mathcal{X}_{\textrm{FPE}}(\omega^{\star})$ for all $l\geq 0$, there exists a positive $c<1$ under 6 such that $\mathcal{C}(\alpha_{l},k)\leq c$, hence:

[TABLE]

Now, for a fixed constant stepsize $\alpha_{l}=\alpha$, we can apply Equation 211 $l$ times, yielding:

[TABLE]

Now we apply the bound $1-x\leq\exp(-x)$, yielding our desired result:

[TABLE]

∎

B.5 Breaking the Deadly Triad

Theorem 4**.**

Let 7 hold over $\mathcal{X}_{\textrm{FPE}}(\omega^{\star})$ from Definition 1. For any $\frac{1}{\alpha_{l}}>\frac{\lambda_{1}^{\min}+\lambda_{1}^{\max}}{2}$ such that $\alpha_{l}>0$, any

[TABLE]

ensures that $\mathcal{X}_{\textrm{FPE}}(\omega^{\star})$ is a region of contraction satisfying 6.

Proof.

Now, as $\left\lvert 1-\alpha_{l}\lambda^{\prime}\right\rvert$ is a symmetric function of $\lambda$ with a minima at $\lambda=\frac{1}{\alpha_{l}}$ and $\frac{\lambda_{1}^{\min}+\lambda_{1}^{\max}}{2}$ is the mid point of $\lambda_{1}^{\min}$ and $\lambda_{1}^{\max}$, it follows:

[TABLE]

Now,

[TABLE]

hence

[TABLE]

Let $\left\lVert\bar{J}_{\textrm{FPE}}^{\star}\right\rVert=1-\epsilon$ where $0<\epsilon<1$. From the definition of a limit, this implies that for $\epsilon$ there exists some finite $k^{\prime}$ such that whenever $k>k^{\prime}$:

[TABLE]

as required. To find the value of $k$ for which $\mathcal{C}(\alpha_{l},k)<1$, we set $\mathcal{C}(\alpha_{l},k)=1$ and solve:

[TABLE]

∎

Appendix C Additional Experiment Information

For both plots, each configuration was run over $5$ random seeds, with the central tendency given by the mean, and the shaded errors representing the standard error of the mean. Hyperparameters that are not varied in the plots were optimised by grid search across either linear or logarithmic hyperparameter ranges, as is suitable. Parameters were chosen that led to the highest performance as averaged across random seeds, then relevant hyperparameters were varied, using the optimal fixed hyperparameters. Hyperparameters that were varied are denoted as lists in the tables below.

C.1 Baird’s Counterexample

Figure Figure 4 shows the counterexample. The behaviour policy chooses between the action represented by the wavy line with probability $6/7$, and the solid line with probability 1/7. The behaviour policy always chooses the solid line. The linear function approximation scheme is shown in terms of the value function weights. Sampling off policy in this way leads to divergence of TD, but PFPE converges, as seen in Figure 2.

C.2 Cartpole Experiment

For the Cartpole experiment, we use a simple DQN-style setup with a small multilayer perceptron (MLP) representing the value function. A small adjustment is made from PFPE as characterised by the paper. Instead of updating value parameters on single data points, parameter updates are averaged across a small batch. This was found to increase stability of learning in both settings, with no notable effects when comparing across independent variables. This means that, in addition to our target network, we also make use of a replay buffer which stores observed transitions. As such, data used in updates was sampled uniformly from previous transitions. The policy was $\epsilon$-greedy, with the estimated optimal action taken with probability $1-\epsilon$. The environment is maintained by OpenAI as part of the gym suite, and falls under MIT licensing.

Appendix D Extensions

As discussed in Section 4, once we can establish 4 then there are several theoretical tools that become applicable from stochastic approximation to prove convergence under a range of assumptions. Brooms (2006) provide a comprehensive overview of classic methods. In particular, stochastic approximation has been shown to converge when sampling from an ergodic Markov chain under specific regularity assumptions (Allasonniere et al., 2010). Perhaps the easiest to verify in our context is those of Andrieu et al. (2005), who provides a series of assumptions that can be checked in practice. Moreover, this theory was recently extended to Markov chains that converge sub-geometrically to their station distributions by Debavelaere et al. (2021). Adherence of the updates to remain in a contractive region can be ensured by projection into an ever increasing subset of $\Omega$ until convergence occurs, which is detailed and analysed in Andradottir (1991).