Expected Policy Gradients

TL;DR

Expected Policy Gradients (EPG) unify stochastic and deterministic policy gradients by integrating over actions, reducing variance, and improving exploration, leading to superior performance in continuous control tasks.

Contribution

EPG introduces a new policy gradient theorem, unifies existing methods, and demonstrates improved exploration and performance without extra computational cost.

Findings

EPG reduces gradient estimate variance.

EPG outperforms DPG with Ornstein-Uhlenbeck noise in MuJoCo tasks.

EPG provides a unified framework for stochastic and deterministic policies.

Abstract

We propose expected policy gradients (EPG), which unify stochastic policy gradients (SPG) and deterministic policy gradients (DPG) for reinforcement learning. Inspired by expected sarsa, EPG integrates across the action when estimating the gradient, instead of relying only on the action in the sampled trajectory. We establish a new general policy gradient theorem, of which the stochastic and deterministic policy gradient theorems are special cases. We also prove that EPG reduces the variance of the gradient estimates without requiring deterministic policies and, for the Gaussian case, with no computational overhead. Finally, we show that it is optimal in a certain sense to explore with a Gaussian policy such that the covariance is proportional to the exponential of the scaled Hessian of the critic with respect to the actions. We present empirical results confirming that this new form of exploration substantially outperforms DPG with the Ornstein-Uhlenbeck heuristic in four challenging MuJoCo domains.

Figures

Tables

Table 1. Table 1: Estimated standard deviation (mean and 90% interval) across runs after learning.

Domain	${\hat{\sigma }}_{\text{DPG}}$	${\hat{\sigma }}_{\text{EPG}}$
HalfCheetah-v1	1336.39 [1107.85, 1614.51]	1056.15 [875.54, 1275.94]
InvertedPendulum-v1	291.26 [241.45, 351.88]	0.00 n/a
Reacher2d-v1	1.22 [0.63, 2.31]	0.13 [0.07, 0.26]
Walker2d-1	543.54 [450.58, 656.65]	762.35 [631.98, 921.00]

Equations

$$\nabla J={\int }_{s}d\rho (s){\int }_{a}d\pi (a\mid s)\nabla \log \pi (a\mid s)(Q(a,s)+b(s)),$$

$$\hat{\nabla }J={\sum }_{t=0}^T{\gamma }^t\nabla \log \pi (a_t\mid s_t)(\hat{Q}(s_t,a_t)+b(s_t)).$$

$$\nabla J={\int }_{s}d\rho (s)\nabla \pi (s){\nabla }_{a}Q(a=\pi (s),s).$$

$$\hat{\nabla }J={\sum }_{t=0}^T{\gamma }^t\nabla \pi (s){\nabla }_a\hat{Q}(a=\pi (s_t),s_t).$$

$$n_i\leftarrow -n_{i-1}\psi +\mathcal{N}(0,\sigma I)a\sim \pi (s)+n_i.$$

$$\hat{Q}(s_t,a_t)\leftarrow$$

$$\displaystyle\;\textstyle\alpha\big{[}r_{t+1}+\gamma\hat{Q}(s_{t+1},a_{t+1})-\hat{Q}(s_{t},a_{t})\big{]}.$$

$$\hat{Q}($$

$$\displaystyle\;\textstyle\;\alpha\big{[}r_{t+1}+\gamma\int_{a}d\pi(a\mid s)\hat{Q}(s_{t+1},a)-\hat{Q}(s_{t},a_{t})\big{]}.$$

$$\hat{\nabla }J={\sum }_{t=0}^T{\gamma }^t\nabla \log \pi (a_t\mid s_t)(r+\gamma \hat{V}(s^{\prime })-\hat{V}(s)),$$

$$\nabla J$$

$$={\int }_{s}d\rho (s)I_{\pi }^Q(s).$$

$$\hat{\nabla }J=\sum _{t=0}^T\underset{g_t}{\underbrace{{\gamma }^t{\hat{I}}_{\pi }^{\hat{Q}}(s_t)}},$$

$$\nabla J={\int }_{s}d\rho (s)\underset{I_G(s)}{\underbrace{\left[\nabla V(s)-{\int }_{a}d\pi (a,s)\nabla Q(a,s)\right]}}.$$

$${\int }_{s}d\rho (s){\int }_{a}d\pi (a,s)\nabla Q(a,s)$$

$$={\int }_{s}d\rho (s){\int }_{a}d\pi (a,s)\nabla (R(a,s)+\gamma {\int }_{s^{\prime }}dp(s^{\prime }\mid s,a)V(s^{\prime }))$$

$$={\int }_{s}d\rho (s){\int }_{a}d\pi (a,s)(\underset{0}{\underbrace{\nabla R(a,s)}}+\gamma {\int }_{s^{\prime }}dp(s^{\prime }\mid s,a)\nabla V(s^{\prime }))$$

$$=\gamma {\int }_{s}d\rho (s){\int }_{s^{\prime }}d{p}_{\pi }(s^{\prime }\mid s)\nabla V(s^{\prime })$$

$$={\int }_{s}d\rho (s)\nabla V(s)-\underset{\nabla J}{\underbrace{{\int }_{s}d{p}_0(s)\nabla V(s)}}$$

$$={\int }_{s}d\rho (s)\nabla V(s)-\nabla J.$$

$$\nabla J$$

$$={\int }_{s}d\rho (s){\int }_{a}d\pi (a\mid s)\nabla \log \pi (a\mid s)Q(a,s).$$

$$\nabla V$$

$${\int }_{a}da(\nabla \pi (a,s))Q(a,s)+{\int }_{a}d\pi (a,s)(\nabla Q(a,s))$$

$$\nabla J={\int }_{s}d\rho (s)I_G(s)={\int }_{s}d\rho (s)\nabla \pi (s){\nabla }_{a}Q(a,s).$$

$$I_G(s)$$

$$=\nabla V(s)-\gamma {\int }_{s^{\prime }}d{p}_{\pi }(s^{\prime }\mid s)\nabla V(s^{\prime })$$

$$=\nabla \pi (s){\nabla }_{a}Q(a,s).$$

$$Q(a,s)$$

$$+\frac{1}{2}(a-{\mu }_s{)}^{\top }H({\mu }_s,s)(a-{\mu }_s)$$

$$=\frac{1}{2}{a}^{\top }H({\mu }_s,s)a+a^{\top }\left({\nabla }_{a}Q(a={\mu }_s,s)-H({\mu }_s,s){\mu }_s\right)+\text{const}.$$

$$I_{\pi (s),{\mu }_s}^Q$$

$$=\nabla {\mu }_s{\nabla }_{a}Q(a={\mu }_s,s)$$

$$I_{\pi (s),{\Sigma }_s^{1/2}}^Q$$

$$I_{\pi }^Q(s)=(\nabla {\mu }_s){\nabla }_{a}Q(a={\mu }_s,s).$$

$${\Sigma }_s^{1/2}\leftarrow {\Sigma }_s^{1/2}+{\alpha }_{n}H(s){\Sigma }_s^{1/2}.$$

$$({\Sigma }_s^{1/2}{)}_n=(I+\alpha H(s){)}^n({\Sigma }_s^{1/2}{)}_0.$$

$$({\Sigma }_s^{1/2}{)}_n=(I+\alpha U\Lambda U^{\top }{)}^n({\Sigma }_s^{1/2}{)}_0=U(I+\alpha \Lambda {)}^n{U}^{\top }({\Sigma }_s^{1/2}{)}_0$$

Full text

Introduction

Policy gradient methods (Sutton et al., 2000; Peters and Schaal, 2006; Peters and Schaal, 2008b; Silver et al., 2014), which optimise policies by gradient ascent, have enjoyed great success in reinforcement learning problems with large or continuous action spaces. The archetypal algorithm optimises an actor, i.e., a policy, by following a policy gradient that is estimated using a critic, i.e., a value function.

The policy can be stochastic or deterministic, yielding stochastic policy gradients (SPG) (Sutton et al., 2000) or deterministic policy gradients (DPG) (Silver et al., 2014). The theory underpinning these methods is quite fragmented, as each approach has a separate policy gradient theorem guaranteeing the policy gradient is unbiased under certain conditions.

Furthermore, both approaches have significant shortcomings. For SPG, variance in the gradient estimates means that many trajectories are usually needed for learning. Since gathering trajectories is typically expensive, there is a great need for more sample efficient methods.

DPG’s use of deterministic policies mitigates the problem of variance in the gradient but raises other difficulties. The theoretical support for DPG is limited since it assumes a critic that approximates $\nabla_{a}Q$ when in practice it approximates $Q$ instead. In addition, DPG learns off-policy111We show in this paper that, in certain settings, off-policy DPG is equivalent to EPG, our on-policy method., which is undesirable when we want learning to take the cost of exploration into account. More importantly, learning off-policy necessitates designing a suitable exploration policy, which is difficult in practice. In fact, efficient exploration in DPG is an open problem and most applications simply use independent Gaussian noise or the Ornstein-Uhlenbeck heuristic (Uhlenbeck and Ornstein, 1930; Lillicrap et al., 2015).

In this paper, we propose a new approach called expected policy gradients (EPG) that unifies policy gradients in a way that yields both theoretical and practical insights. Inspired by expected sarsa (Sutton and Barto, 1998; van Seijen et al., 2009), the main idea is to integrate across the action selected by the stochastic policy when estimating the gradient, instead of relying only on the action selected during the sampled trajectory.

EPG enables two theoretical contributions. First, we establish a number of equivalences between EPG and DPG, among which is a new general policy gradient theorem, of which the stochastic and deterministic policy gradient theorems are special cases. Second, we prove that EPG reduces the variance of the gradient estimates without requiring deterministic policies and, for the Gaussian case, with no computational overhead over SPG.

EPG also enables a practical contribution: a principled exploration strategy for continuous problems. We show that it is optimal in a certain sense to explore with a Gaussian policy such that the covariance is proportional to $e^{H}$, where $H$ is the scaled Hessian of the critic with respect to the actions. We present empirical results confirming that this new approach to exploration substantially outperforms DPG with Ornstein-Uhlenbeck exploration in four challenging MuJoCo domains.

Background

A Markov decision process is a tuple $(S,A,R,p,p_{0},\gamma)$ where $S$ is a set of states, $A$ is a set of actions (in practice either $A=\mathbb{R}^{d}$ or $A$ is finite), $R(s,a)$ is a reward function, $p(s^{\prime}\mid a,s)$ is a transition kernel, $p_{0}$ is an initial state distribution, and $\gamma\in[0,1)$ is a discount factor. A policy $\pi(a\mid s)$ is a distribution over actions given a state. We denote trajectories as $\tau^{\pi}=(s_{0},a_{0},r_{0},s_{1},a_{1},r_{1},\dots)$, where $s_{0}\sim p_{0}$, $a_{t}\sim\pi(\cdot\mid s_{t-1})$ and $r_{t}$ is a sample reward. A policy $\pi$ induces a Markov process with transition kernel $p_{\pi}(s^{\prime}\mid s)=\int_{a}d\pi(a\mid s)p(s^{\prime}\mid a,s)$ where we use the symbol $d\pi(a\mid s)$ to denote Lebesgue integration against the measure $\pi(a\mid s)$ where $s$ is fixed. We assume the induced Markov process is ergodic with a single invariant measure defined for the whole state space. The value function is $V^{\pi}={\mathbb{E}_{\tau}}\left[\sum_{i}\gamma_{i}r_{i}\right]$ where actions are sampled from $\pi$. The $Q$-function is $Q^{\pi}(a\mid s)={\mathbb{E}_{R}}\left[r\,\middle|\,s,a\right]+\gamma{\mathbb{E}_{p(s\ \mid s)}}\left[V^{\pi}(s^{\prime})\,\middle|\,s\right]$ and the advantage function is $A^{\pi}(a\mid s)=Q^{\pi}(a\mid s)-V^{\pi}(s)$. An optimal policy maximises the total return $J=\int_{s}dp_{0}(s)V^{\pi}(s)$. Since we consider only on-policy learning with just one current policy, we drop the $\pi$ super/subscript where it is redundant.

If $\pi$ is parameterised by $\theta$, then stochastic policy gradients (SPG) (Sutton et al., 2000; Peters and Schaal, 2006; Peters and Schaal, 2008b) perform gradient ascent on $\nabla J$, the gradient of $J$ with respect to $\theta$ (gradients without a subscript are always with respect to $\theta$). For stochastic policies, we have:

[TABLE]

where $\rho$ is the discounted-ergodic occupancy measure, defined in the supplement, and $b(s)$ is a baseline, which can be any function that depends on the state but not the action, since $\int_{a}d\pi(a\mid s)\nabla\log\pi(a\mid s)b(s)=0$. Typically, (1) is approximated from samples from a trajectory $\tau$ of length $T$:

[TABLE]

If the policy is deterministic (we denote it $\pi(s)$), we can use deterministic policy gradients (Silver et al., 2014) instead:

[TABLE]

This update is then approximated using samples:

[TABLE]

Since the policy is deterministic, the problem of exploration is addressed using an external source of noise, typically modeled using a zero-mean Ornstein-Uhlenbeck (OU) process (Uhlenbeck and Ornstein, 1930; Lillicrap et al., 2015) parametrized by $\psi$ and $\sigma$:

[TABLE]

In (2) and (4), $\hat{Q}$ is a critic that approximates $Q$ and can be learned by sarsa (Rummery and Niranjan, 1994; Sutton, 1996):

[TABLE]

Alternatively, we can use expected sarsa (Sutton and Barto, 1998; van Seijen et al., 2009), which marginalises out $a_{t+1}$, the distribution over which is specified by the known policy, to reduce the variance in the update:

[TABLE]

We could also use advantage learning (Baird and others, 1995) or LSTDQ (Lagoudakis and Parr, 2003). If the critic’s function approximator is compatible, then the actor, i.e., $\pi$, converges (Sutton et al., 2000).

Instead of learning $\hat{Q}$, we can set $b(s)=-V(s)$ so that $Q(a,s)+b(s)=A(s,a)$ and then use the TD error $\delta(r,s^{\prime},s)=r+\gamma V(s^{\prime})-V(s)$ as an estimate of $A(s,a)$ (Bhatnagar et al., 2008):

[TABLE]

where $\hat{V}(s)$ is an approximate value function learned using any policy evaluation algorithm. (8) works because ${\mathbb{E}}\left[\delta(r,s^{\prime},s)\,\middle|\,a,s\right]=A(s,a)$, i.e., the TD error is an unbiased estimate of the advantage function. The benefit of this approach is that it is sometimes easier to approximate $V$ than $Q$ and that the return in the TD error is unprojected, i.e., it is not distorted by function approximation. However, the TD error is noisy, introducing variance in the gradient.

To cope with this variance, we can reduce the learning rate when the variance of the gradient would otherwise explode, using, e.g., Adam (Kingma and Ba, 2014), natural policy gradients (Kakade, 2002; Amari, 1998; Peters and Schaal, 2008a), the adaptive step size method (Pirotta, Restelli, and Bascetta, 2013) or Newton’s method (Furmston and Barber, 2012; Parisi, Pirotta, and Restelli, 2016). However, this results in slow learning when the variance is high. One can also use PGPE, which replaces the stochastic policy with a distribution over deterministic policies (Sehnke et al., 2010). However, PGPE precludes updating the current policy during the episode and makes it difficult to explore efficiently.

We can also eliminate all variance caused by the policy at the cost of making the policy deterministic and using the DPG update, which usually necessitates performing off-policy exploration. EPG, presented below, reduces to DPG in many useful cases, while providing a principled way to explore and also allowing for stochastic policies.

Yet another way to eliminate variance in the actor is not to have an actor at all, instead selecting actions soft-greedily with respect to $\hat{Q}$ learned using sarsa. This is trivial for discrete actions and can also be done with a one-step Newton’s method for $Q$-functions that are quadric in the actions (Gu et al., 2016b).

Expected Policy Gradients

In this section, we propose expected policy gradients (EPG).

Main Algorithm

First, we introduce $I^{Q}_{\pi}(s)$ to denote the inner integral in (1):

[TABLE]

This suggests a new way to write the approximate gradient:

[TABLE]

where $\hat{I}^{\hat{Q}}_{\pi}(s)$ is some approximation to $I^{\hat{Q}}_{\pi}(s)=\int_{a}d\pi(a\mid s)\nabla\log\pi(a\mid s)(\hat{Q}(a,s)+b(s))$. This approach makes explicit that one step in estimating the gradient is to evaluate an integral to estimate $I^{\hat{Q}}_{\pi}(s)$. The main insight behind EPG is that, given a state, $I^{\hat{Q}}_{\pi}(s)$ is expressed fully in terms of known quantities. Hence we can manipulate it analytically to obtain a formula or we can just compute the integral using any numerical quadrature if an analytical solution is impossible.

SPG as given in (2) performs this quadrature using a simple one-sample Monte Carlo method. However, relying on such a method is unnecessary. In fact, the actions used to interact with the environment need not be used at all in the evaluation of $\hat{I}_{\pi}^{Q}(s)$ since $a$ is a bound variable in the definition of $I_{\pi}^{Q}(s)$. The motivation is thus similar to that of expected sarsa but applied to the actor’s gradient estimate instead of the critic’s update rule. EPG, shown in Algorithm 1, uses (10) to form a policy gradient algorithm that repeatedly estimates $\hat{I}_{\pi}^{Q}(s)$ with an integration subroutine.

EPG has benefits even when an analytical solution is not possible: if the action space is low dimensional, numerical quadrature is cheap; if it is high dimensional, it is still often worthwhile to balance the expense of simulating the system with the cost of quadrature. Actually, even in the extreme case of expensive quadrature but cheap simulation, the limited resources available for quadrature could still be better spent on EPG with smart quadrature than SPG with simple Monte Carlo. One of the motivations of DPG was precisely that the simple one-sample Monte-Carlo quadrature implicitly used by SPG often yields high variance gradient estimates, even with a good baseline. To see why, consider Figure 1 (left). A simple Monte Carlo method evaluates the integral by sampling one or more times from $\pi(a\mid s)$ (blue) and evaluating $\nabla_{\mu}\log\pi(a\mid s)Q(a,s)$ (red) as a function of $a$. A baseline can decrease the variance by adding a multiple of $\nabla_{\mu}\log\pi(a\mid s)$ to the red curve, but the problem remains that the red curve has high values where the blue curve is almost zero. Consequently, substantial variance persists, whatever the baseline, even with a simple linear $Q$-function, as shown in Figure 1 (right). DPG addressed this problem for deterministic policies but EPG extends it to stochastic ones.

Relationship to Other Methods

EPG has some similarities with VINE sampling (Schulman et al., 2015), which uses an (intrinsically noisy) Monte Carlo quadrature with many samples.222VINE sampling also differs from EPG by performing independent rollouts of $Q$, requiring a simulator with reset. However, the example in Figure 1 shows that even with a computationally expensive many-sample Monte Carlo method, the problem of variance remains, regardless of the baseline.

EPG is also related to variance minimisation techniques that interpolate between two estimators, e.g., (Gu et al., 2016a, Eq. 7) is similar to Corollary 4. However, EPG uses a quadric (not linear) approximation to the critic, which is crucial for exploration. Furthermore, it completely eliminates variance in the inner integral, as opposed to just reducing it.

The idea behind EPG was also independently and concurrently developed as Mean Actor Critic (Asadi et al., 2017), though only for discrete actions and without a supporting theoretical analysis.

Gaussian Policies

EPG is particularly useful when we make the common assumption of a Gaussian policy: we can then perform the integration analytically under reasonable conditions. We show below (see Lemma 3) that the update to the policy mean computed by EPG is equivalent to the DPG update. Moreover, a simple formula for the covariance can be derived (see Lemma 2). Algorithms 2 and 3 show the resulting special case of EPG, which we call Gaussian policy gradients (GPG).

Surprisingly, GPG is on-policy but nonetheless fully equivalent to DPG, an off-policy method, with a particular form of exploration. Hence, GPG, by specifying the policy’s covariance, can be seen as a derivation of an exploration strategy for DPG. In this way, GPG addresses an important open question. As we show later, this leads to improved performance in practice.

The computational cost of GPG is small: while it must store a Hessian matrix $H(a,s)=\nabla^{2}_{a}\hat{Q}(a,s)$, its size is only $d\times d$, where $A=\mathbb{R}^{d}$, which is typically small, e.g., $d=6$ for HalfCheetah-v1. This Hessian is the same size as the policy’s covariance matrix, which any policy gradient must store anyway, and should not be confused with the Hessian with respect to the parameters of the neural network, as used with Newton’s or natural gradient methods (Peters and Schaal, 2008a; Furmston, Lever, and Barber, 2016), which can easily have thousands of entries. Hence, GPG obtains EPG’s variance reduction essentially for free.

Analysis

In this section, we analyse EPG, showing that it unifies SPG and DPG, that $\hat{I}_{\pi}^{Q}(s)$ can often be computed analytically, and that EPG has lower variance than SPG.

General Policy Gradient Theorem

We begin by stating our most general result, showing that EPG can be seen as a generalisation of both SPG and DPG. To do this, we first state a new general policy gradient theorem. We use the shorthand $\nabla$ without a subscript to denote the gradient with respect to policy parameters $\theta$.

Theorem 1** (General Policy Gradient Theorem).**

If $\pi(\cdot,s)$ is a normalised Lebesgue measure for all $s$, then

[TABLE]

Proof.

We begin by expanding the following expression.

[TABLE]

The first equality follows by expanding the definition of $Q$ and the penultimate one follows from Lemma B (in the supplement). Then the theorem follows by rearranging terms. ∎

The crucial benefit of Theorem 1 is that it works for all policies, both stochastic and deterministic, unifying previously separate derivations for the two settings. To show this, in the following two corollaries, we use Theorem 1 to recover the stochastic policy gradient theorem (Sutton et al., 2000) and the deterministic policy gradient theorem (Silver et al., 2014), in each case by introducing additional assumptions to obtain a formula for $I_{G}(s)$ expressible in terms of known quantities.

Corollary 1** (Stochastic Policy Gradient Theorem).**

If $\pi(\cdot\mid s)$ is differentiable, then

[TABLE]

Proof.

We obtain the following by expanding $\nabla V$.

[TABLE]

We obtain $I_{G}(s)=\int_{a}d\pi(a\mid s)\nabla\log\pi(a\mid s)Q(a,s)=I^{Q}_{\pi}(s)$ by plugging this into the definition of $I_{G}(s)$. We obtain $\nabla J$ by invoking Theorem 1 and plugging in the above expression for $I_{G}(s)$. ∎

We now recover the DPG update introduced in (3).

Corollary 2** (Deterministic Policy Gradient Theorem).**

If $\pi(\cdot\mid s)$ is a Dirac-delta measure (i.e., a deterministic policy) and $Q(\cdot,s)$ is differentiable, then

[TABLE]

Proof.

We begin by obtaining an expression for $I_{G}(s)$.

[TABLE]

Here, the second equality follows by expanding the definition of $Q$ and the third follows from an established deterministic policy gradient result (Silver et al., 2014, Supplement, Eq. 1). We can then obtain $\nabla J$ by invoking Theorem 1 and plugging in the above expression for $I_{G}(s)$. ∎

These corollaries show that the choice between deterministic and stochastic policy gradients is fundamentally a choice of quadrature method. Hence, the empirical success of DPG relative to SPG (Silver et al., 2014; Lillicrap et al., 2015) can be understood in a new light. In particular, it can be attributed, not to a fundamental limitation of stochastic policies (indeed, stochastic policies are sometimes preferred), but instead to superior quadrature. DPG integrates over Dirac-delta measures, which is known to be easy, while SPG typically relies on simple Monte Carlo integration. Thanks to EPG, a deterministic approach is no longer required to obtain a method with low variance.

We add as a sideline that since Theorem 1 can be written as $I_{G}(s)=\nabla V(s)-\gamma\int_{s^{\prime}}dp_{\pi}(s^{\prime}\mid s)\nabla V(s^{\prime})$, which involves the derivatives of value functions, GPG resembles value gradients (Heess et al., 2015). However, in our case, we are learning $\nabla J$ directly and do not perform recursive estimation of $\nabla V$ as value gradient methods do.

Analytical Quadrature - Gaussian Policy

We now derive a lemma supporting GPG.

Lemma 1** (Gaussian Policy Gradients).**

If the policy is Gaussian, i.e. $\pi(\cdot|s)\sim\mathcal{N}(\mu_{s},\Sigma_{s})$ with $\mu_{s}$ and $\Sigma^{1/2}_{s}$ parametrised by $\theta$, where $\Sigma^{1/2}_{s}$ is symmetric and $\Sigma^{1/2}_{s}\Sigma^{1/2}_{s}=\Sigma_{s}$ and the critic is of the form $Q(a,s)=a^{\top}A(s)a+a^{\top}B(s)+\text{const}$ where $A(s)$ is symmetric for every $s$, then $I^{Q}_{\pi}(s)=I^{Q}_{\pi(s),\mu_{s}}+I^{Q}_{\pi(s),\Sigma^{1/2}_{s}}$, where the mean and covariance components are given by $I^{Q}_{\pi(s),\mu_{s}}=(\nabla\mu_{s})(2A(s)\mu+B(s))$ and $I^{Q}_{\pi(s),\Sigma^{1/2}_{s}}=(\nabla\Sigma^{1/2}_{s})2A(s)\Sigma^{1/2}_{s}$.

See Lemma 1 in the supplement for proof of this result. While Lemma 1 requires the critic to be quadric in the actions, this assumption is not very restrictive since the coefficients $B(s)$ and $A(s)$ can be arbitrary continuous functions of the state, e.g., a neural network.

Arbitrary Critics

If $Q$ does not meet the conditions of Lemma 1, we can approximate $Q$ with a quadric function in the neighbourhood of the policy mean. This approximation is motivated by two arguments. First, in MDPs that model physical systems with reasonable reward functions, $Q$ is fairly smooth. Second, policy gradients are a local, incremental method anyway – since the policy mean changes slowly, the values of $Q$ for actions far from the policy mean are usually not relevant for the current update.

Corollary 3** (Approximate Gaussian Policy Gradients with an Arbitrary Critic).**

If the policy is Gaussian, i.e. $\pi(\cdot|s)\sim\mathcal{N}(\mu_{s},\Sigma_{s})$ with $\mu_{s}$ and $\Sigma^{1/2}_{s}$ parametrised by $\theta$ as in Lemma 1 and any critic $Q(a,s)$ doubly differentiable with respect to actions for each state, then $I^{Q}_{\pi(s),\mu_{s}}\approx(\nabla\mu_{s})\nabla_{a}Q(a=\mu_{s},s)$ and $I^{Q}_{\pi(s),\Sigma^{1/2}_{s}}\approx(\nabla\Sigma^{1/2}_{s})H(\mu_{s},s)\Sigma^{1/2}_{s}$, where $H(\mu_{s},s)$ is the Hessian of $Q$ with respect to $a$, evaluated at $\mu_{s}$ for a fixed $s$.

Proof.

We begin by approximating the critic (for a given $s$) using the first two terms of the Taylor expansion of $Q$ in $\mu_{s}$.

[TABLE]

Because of the series truncation, the function on the righthand side is quadric and we can then use Lemma 1:

[TABLE]

∎

To actually obtain the Hessian, we could use automatic differentiation to compute it analytically. Alternatively, we can observe that, if the critic really is quadric, we can just read off the coefficients of the quadric term directly. Therefore, we can approximate the Hessian by generating a number of random action-values around $\mu_{s}$, computing the $Q$ values, and (locally) fitting a quadric. This process is typically more computationally expensive than automatic differentiation but has the advantage of working with ReLU networks (where the true Hessian is zero but we still have a kind of global curvature after smoothing) and leveraging more information from the critic (since the evaluation is at more than one point).

Linear GPG

We now state a consequence of Lemma 1 for the case when the critic $Q$ is linear in the actions, i.e., the quadric term is always zero.

Corollary 4** (Linear Gaussian Policy Gradients).**

If the policy is Gaussian, i.e., $\pi(\cdot|s)\sim\mathcal{N}(\mu_{s},\Sigma_{s})$ with $\mu_{s}$ parametrised by $\theta$ and the critic is of the form $Q(a\mid s)=a^{\top}B(s)+\text{const}$, then $I^{Q}_{\pi}(s)=(\nabla\mu_{s})B(s)$. Moreover, it is unnecessary to parameterise $\Sigma^{1/2}_{s}$ since the policy gradient w.r.t. to $\Sigma^{1/2}_{s}$ is zero (i.e., a linear $Q$-function does not give any information about the exploration covariance).

We make Corollary 4 explicit for two reasons. First, it is useful for showing an equivalence between DPG and EPG (see below). Second, it may actually be useful for a non-trivial class of physical systems: if the time-sampling frequency is high enough (which implies acting in small steps), the critic is effectively only used to say if a small step one way is preferable to small step the other way – a linear property.

Equivalences between EPG and DPG

The update for the policy mean obtained in Corollary 3 is the same as the DPG update, linking the two methods:

[TABLE]

We now formalise the equivalences between EPG and DPG. First, on-policy GPG with a linear critic (or an arbitrary critic approximated by the first term in the Taylor expansion) is equivalent to DPG with a Gaussian exploration policy where the covariance stays the same. This follows from Corollary 4. Second, on-policy GPG with a quadric critic (or an arbitrary critic approximated by the first two terms in the Taylor expansion) is equivalent to DPG with a Gaussian exploration policy where the covariance is computed using the update (where $\alpha_{n}$ is a sequence of step-sizes):

[TABLE]

This follows from Corollary 3. Third, and most generally, for any critic at all (not necessarily quadric), DPG is a kind of EPG for a particular choice of quadrature (using a Dirac measure). This follows from Theorem 1.

Surprisingly, this means that DPG, normally considered to be off-policy, can also be seen as on-policy when exploring with Gaussian noise. Furthermore, the compatible critic for DPG (Silver et al., 2014) is indeed linear in the actions. Hence, this relationship holds whenever DPG uses a compatible critic.333The notion of compatibility of a critic is different for stochastic and deterministic policy gradients. Furthermore, Lemma 1 lends new legitimacy to the common practice of replacing the critic required by the DPG theory, which approximates $\nabla_{a}Q$, with one that approximates $Q$ itself, as done in SPG and EPG.

Exploration using the Hessian

The second equivalence given above suggests that we can include the covariance in the actor network and learn it along with the mean. However, another option is to compute it from scratch at each iteration by analytically computing the result of applying (11) infinitely many times.

Lemma 2** (Exploration Limit).**

The iterative procedure defined by the equation $\Sigma^{1/2}_{s}\leftarrow\Sigma^{1/2}_{s}+\alpha H(s)\Sigma^{1/2}_{s}$ applied $n$ times using the diminishing learning rate $\alpha=1/n$ converges to $\Sigma^{1/2}_{s}\propto e^{H(s)}$ as $n\rightarrow\infty$.

Proof.

Consider the sequence $(\Sigma^{1/2}_{s})_{0}=\sigma_{0}I$, $(\Sigma^{1/2}_{s})_{n}=(\Sigma^{1/2}_{s})_{n-1}+\alpha H(s)(\Sigma^{1/2}_{s})_{n-1}$. Expanding out the recursion, the $n$-th element of the sequence is given as:

[TABLE]

We diagonalise the Hessian as $H(s)=U\Lambda U^{\top}$ for some orthonormal matrix $U$ and obtain the following expression for $(\Sigma^{1/2}_{s})_{n}$.

[TABLE]

Since we have $\lim_{n\rightarrow\infty}(1+\frac{1}{n}\lambda)^{n}=e^{\lambda}$ for each diagonal entry of $\Lambda$, we plug $\alpha=\frac{1}{n}$ and obtain the identity:

[TABLE]

∎

The practical implication of Lemma 2 is that, in a policy gradient method, it is justified to use Gaussian exploration with covariance proportional to $e^{cH}$ for some reward scaling constant $c$. Thus by exploring with (scaled) covariance $e^{cH}$, we obtain a principled alternative to the Ornstein-Uhlenbeck heuristic defined in (5). Our results below show that it also performs much better in practice.

Lemma 2 has an intuitive interpretation. If $H(s)$ has a large positive eigenvalue $\lambda$, then $\hat{Q}(s,\cdot)$ has a sharp minimum along the corresponding eigenvector, and the corresponding eigenvalue of $\Sigma$ is $e^{\lambda}$, i.e., also large. The result is a large exploration bonus along that direction, enabling the algorithm to leave local minima. Conversely, if $\lambda$ is negative, then $\hat{Q}(s,\cdot)$ has a maximum and so $e^{\lambda}$ is small, since exploration is not needed.

Variance Analysis

We now prove that for any policy, the EPG estimator of (10) has lower variance than the SPG estimator of (2).

Lemma 3**.**

If for all $s\in S$, the random variable $\nabla\log\pi(a\mid s)\hat{Q}(s,a)$ where $a\sim\pi(\cdot|s)$ has nonzero variance, then

[TABLE]

The proof is deferred to the supplement (see Lemma 3 there). Lemma 3’s assumption is reasonable since the only way a random variable $\nabla\log\pi(a\mid s)\hat{Q}(s,a)$ could have zero variance is if it were the same for all actions in the policy’s support (except for sets of measure zero), in which case optimising the policy would be unnecessary. Since we know that both the estimators of (2) and (10) are unbiased, the estimator with lower variance has lower MSE.

Extension to Entropy Regularisation

On-policy SPG sometimes includes an entropy term in the gradient in order to aid exploration by making the policy more stochastic. The gradient of the differential entropy444For discrete action spaces, the same derivation with integrals replaced by sums holds for the entropy. $\mathcal{H}(s)$ of the policy at state $s$ is defined as follows.

[TABLE]

Typically, we weight the entropy update with the policy gradient update:

[TABLE]

This equation makes clear that performing entropy regularisation is equivalent to using a different critic with $Q$-values shifted by $\alpha\log\pi(a|s)$; this holds for both SPG and EPG.

Experiments

While EPG has many potential uses, we focus on empirically evaluating one particular application: exploration driven by the Hessian exponential (as introduced in Algorithm 2 and Lemma 2), replacing the standard Ornstein-Uhlenbeck (OU) exploration in continuous action domains. To this end, we applied EPG to four domains modelled with the MuJoCo physics simulator (Todorov, Erez, and Tassa, 2012): HalfCheetah-v1, InvertedPendulum-v1, Reacher2d-v1 and Walker2d-v1 and compared its performance to DPG and SPG.

In practice, EPG differed from deep DPG (Lillicrap et al., 2015; Silver et al., 2014) only in the exploration strategy, though their theoretical underpinnings are different. The hyperparameters for DPG and those of EPG that are not related to exploration were taken from an existing benchmark (Islam et al., 2017; Brockman et al., 2016). The exploration hyperparameters for EPG were $\sigma_{0}^{2}=0.2$ and $c=1.0$ where the exploration covariance is $\sigma_{0}^{2}e^{cH}$. These values were obtained using a grid search from the set $\{0.2,0.5,1\}$ for $\sigma_{0}^{2}$ and $\{0.5,1.0,2.0\}$ for $c$ over the HalfCheetah-v1 domain. Since $c$ is just a constant scaling the rewards, it is reasonable to set it to $1.0$ whenever reward scaling is already used. Hence, our exploration strategy has just one hyperparameter $\sigma_{0}^{2}$ as opposed specifying a pair of parameters (standard deviation and mean reversion constant) for OU. We used the same learning parameters for the other domains. For SPG555We tried learning the covariance for SPG but the covariance estimate was unstable; no regularisation hyperparameters we tested matched SPG’s performance with OU even on the simplest domain., we used OU exploration and a constant diagonal covariance of $0.2$ in the actor update (this approximately corresponds to the average variance of the OU process over time). The other parameters for SPG are the same as for the rest of the algorithm. For the learning curves, we obtained 90% confidence intervals around the learning curves. The learning curves show results of independent evaluation runs which used actions generated by the policy mean without any exploration noise.

The results (Figure 2) show that EPG’s exploration strategy yields much better performance than DPG with OU. Furthermore, SPG does poorly, solving only the easiest domain (InvertedPendulum-v1) reasonably quickly, achieving slow progress on HalfCheetah-v1, and failing entirely on the other domains. This is not surprising DPG was introduced precisely to solve the problem of high variance SPG estimates on this type of problem. In InvertedPendulum-v1, SPG initially learns quickly, outperforming the other methods. This is because noisy gradient updates provide a crude, indirect form of exploration that happens to suit this problem. Clearly, this is inadequate for more complex domains: even for this simple domain it leads to subpar performance late in learning.

In addition, EPG typically learns more consistently than DPG with OU. In two tasks, the empirical standard deviation across runs of EPG ($\hat{\sigma}_{\text{EPG}}$) was substantially lower than that of DPG ($\hat{\sigma}_{\text{DPG}}$) at the end of learning, as shown in Table 1. For the other two domains, the confidence intervals around the empirical standard deviations for DPG and EPG were too wide to draw conclusions.

Surprisingly, for InvertedPendulum-v1, DPG’s learning curve declines late in learning. The reason can be seen in the individual runs shown in Figure 3: both DPG and SPG suffer from severe unlearning. This unlearning cannot be explained by exploration noise since the evaluation runs just use the mean action, without exploring. Instead, OU exploration in DPG may be too coarse, causing the optimiser to exit good optima, while SPG unlearns due to noise in the gradients. The noise also helps speed initial learning, as described above, but this does not transfer to other domains. EPG avoids this problem by automatically reducing the noise when it finds a good optimum, i.e., a Hessian with large negative eigenvalues.

Conclusions

This paper proposed a new policy gradient method called expected policy gradients (EPG), that integrates across the action selected by the stochastic policy. We used EPG to prove a new general policy gradient theorem subsuming the stochastic and deterministic policy gradient theorems. We also showed that, under certain realistic conditions, the quadrature required by EPG can be performed analytically, allowing DPG with principled exploration. We presented empirical results confirming that this application of EPG outperforms DPG and SPG on four domains.

Acknowledgements

This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement number 637713).

Supplement

We first provide formal proofs for certain statements invoked by our paper. We then provide a brief discussion of the use of a learning rate that diminished in the trajectory length in the computation of the covariance.

Proofs

First, we prove two lemmas concerning the discounted-ergodic measure $\rho(s)$ which have been implicitly realised for some time but as far as we could find, never proved explicitly.

Definition 1** (Time-dependent occupancy).**

[TABLE]

Definition 2** (Truncated trajectory).**

Define the trajectory truncated after $N$ steps as $\tau^{N}=(s_{0},a_{0},r_{0},s_{1},a_{1},r_{1},\dots,s_{N})$.

Observation 1** (Expectation wrt. truncated trajectory).**

Since $\tau_{N}=(s_{0},s_{1},s_{2},\dots,s_{N})$ is associated with the density $\prod_{i=0}^{N-1}p(s_{i+1}\mid s_{i})p_{0}(s_{0})$, we have that

[TABLE]

for any function $f$.

Definition 3** (Expectation with respect to infinte trajectory).**

For any bounded function $f$, we have

[TABLE]

Here, the sum on the left-hand side is part of the symbol being defined.

Observation 2** (Property of expectation with respect to infinte trajectory).**

[TABLE]

for any bounded function $f$.

Definition 4** (Discounted-ergodic occupancy measure $\rho$).**

[TABLE]

The measure $\rho$ is not normalised in general. Intuitively, it can be thought of as ‘marginalising out’ the time in the system dynamics.

Lemma 4** (Discounted-ergodic property).**

For any bounded function $f$:

[TABLE]

Proof.

[TABLE]

Here, the first equality follows from Observation 2. ∎

This property is useful since the expression on the left can be easily manipulated while the expression on the right can be estimated from samples using Monte Carlo.

Lemma 5** (Generalised eigenfunction property).**

For any bounded function $f$:

[TABLE]

Proof.

[TABLE]

Here, the first equality follows form definition 4, the second one from definition 1. The last equality follows again from definition 4. ∎

Definition 5** (Markov Reward Process).**

A Markov Reward Process is a tuple $(p,p_{0},R,\gamma)$, where $p(s^{\prime}|s)$ is a transition kernel, $p_{0}$ is the distribution over initial states, $R(\cdot|s)$ is a reward distribution conditioned on the state and $\gamma$ is the discount constant.

An MRP can be thought of as an MDP with a fixed policy and dynamics given by marginalising out the actions $p_{\pi}(s^{\prime}\mid s)=\int_{a}d\pi(a\mid s)p(s^{\prime}\mid a,s)$. Since this paper considers the case of one policy, we abuse notation slightly by using the same symbol $\tau$ to denote trajectories including actions, i.e. $(s_{0},a_{0},r_{0},s_{1},a_{1},r_{1},\dots)$ and without them $(s_{0},r_{0},s_{1},r_{1},\dots)$.

Lemma 6** (Second Moment Bellman Equation).**

Consider a Markov Reward Process $(p,p_{0},X,\gamma)$ where $p(s^{\prime}\mid s)$ is a Markov process and $X(\cdot\mid s)$ is some probability density function666Note that while $X$ occupies a place in the definition of the MRP usually called ‘reward distribution’, we are using the symbol $X$, not $R$ since we shall apply the lemma to $X$es which are constructions distinct from the reward of the MDP we are solving.. Denote the value function of the MRP as $V$. Denote the second moment function $S$ as

[TABLE]

Then $S$ is the value function of the MRP: $(p,p_{0},u,\gamma^{2})$, where $u(s)$ is a deterministic random variable given by

[TABLE]

Proof.

[TABLE]

This is exactly the Bellman equation of the MRP $(p,p_{0},u,\gamma^{2})$. The theorem follows since the Bellman equation uniquely determines the value function. ∎

Observation 3** (Dominated Value Functions).**

Consider two Markov Reward Processes $(p,p_{0},X_{1},\gamma)$ and $(p,p_{0},X_{2},\gamma)$, where $p(s^{\prime}\mid s)$ is a Markov process (common to both MRPs) and $X_{1}(s)$, $X_{2}(s)$ are some deterministic random variables meeting the condition $X_{1}(s)\leq X_{2}(s)$ for every $s$. Then the value functions $V_{1}$ and $V_{2}$ of the respective MRPs satisfy $V_{1}(s)\leq V_{2}(s)$ for every $s$. Moreover, if we have that $X_{1}(s)<X_{2}(s)$ for all states, then the inequality between value functions is strict.

Proof.

Follows trivially by expanding the value function as a series and comparing series elementwise. ∎

We now move our attention to prove the Gaussian Policy Gradients lemma.

Lemma 1** (Gaussian Policy Gradients).**

If the policy is Gaussian, i.e. $\pi(\cdot|s)\sim\mathcal{N}(\mu_{s},\Sigma_{s})$ with $\mu_{s}$ and $\Sigma^{1/2}_{s}$ parametrised by $\theta$, where $\Sigma^{1/2}_{s}$ is symmetric and $\Sigma^{1/2}_{s}\Sigma^{1/2}_{s}=\Sigma_{s}$ and the critic is of the form $Q(a,s)=a^{\top}A(s)a+a^{\top}B(s)+\text{const}$ where $A(s)$ is symmetric for every $s$, then $I^{Q}_{\pi}(s)=I^{Q}_{\pi(s),\mu_{s}}+I^{Q}_{\pi(s),\Sigma^{1/2}_{s}}$, where the mean and covariance components are given by $I^{Q}_{\pi(s),\mu_{s}}=(\nabla\mu_{s})(2A(s)\mu+B(s))$ and $I^{Q}_{\pi(s),\Sigma^{1/2}_{s}}=(\nabla\Sigma^{1/2}_{s})2A(s)\Sigma^{1/2}_{s}$.

Proof.

First, we observe that the critic $Q$ defined in the statement of the lemma does not depend on the policy parameters $\theta$. This is because $Q$ is an approximation to the Q-function maintained by the algorithm as opposed to the true Q-function, which is defined with respect to the policy and does depend on it.

We can hence move the differentiation outside of the integral, as follows.

[TABLE]

We now expand the expectation using a known expression for the expectation of a a quadratic form:

[TABLE]

This gives way to the following derivatives.

[TABLE]

.

We now obtain the result by applying chain rule.

[TABLE]

∎

Lemma 3**.**

If for all $s\in S$, the random variable $\nabla\log\pi(a\mid s)\hat{Q}(s,a)$ where $a\sim\pi(a|s)$ has nonzero variance, then

[TABLE]

Proof.

Both random variables have the same mean so we need only show that:

[TABLE]

We start by applying Lemma 6 to the lefthand side and setting $X=X_{1}(s_{t})=\gamma^{t}\nabla\log\pi(a_{t}\mid s_{t})(\hat{Q}(s_{t},a_{t})+b(s_{t}))$ where $a_{t}\sim\pi(a_{t}|s_{t})$. This shows that ${\mathbb{E}_{\tau}}\left[\left(\sum_{t=0}^{\infty}\gamma^{t}\nabla\log\pi(a_{t}\mid s_{t})(\hat{Q}(s_{t},a_{t})+b(s_{t}))\right)^{2}\right]$ is the total return of the MRP $(p,p_{0},u_{1},\gamma^{2})$, where

[TABLE]

Likewise, applying Lemma 6 again to the righthand side, instantiating $X$ as a deterministic random variable $X_{2}(s_{t})=I^{\hat{Q}}_{\pi}(s_{t})$, we have that ${\mathbb{E}_{\tau}}\left[\sum_{t=0}^{\infty}\left(\gamma^{t}I^{\hat{Q}}_{\pi}(s_{t})\right)^{2}\right]$ is the total return of the MRP $(p,p_{0},u_{2},\gamma^{2})$, where

[TABLE]

Note that ${\mathbb{E}_{X_{1}(x\mid s)}}\left[x\right]={\mathbb{E}_{X_{2}(x\mid s)}}\left[x\right]$ and therefore $u_{1}\geq u_{2}$. Furthermore, by assumption of the lemma, the inequality is strict. The lemma then follows by applying Observation 3. ∎

For convenience, Lemma 3 also assumes infinite length trajectories. However, this is not a practical limitation since all policy gradient methods implicitly assume trajectories are long enough to be modelled as infinite. Furthermore, a finite trajectory variant also holds, though the proof is messier.

Remarks on the covariance limit

When we obtain $e^{H}$ as the limiting covariance matrix in Lemma 2 of the main paper, there is a slight modelling difficulty: is it justified to use the learning rate of $\frac{1}{n}$, which diminished in the length of the trajectory, as opposed to a small finite number? We observe that the problem of choosing step sizes is, in general, not specific to our method since all policy gradient methods rely on stochastic optimisation and hence work with a diminishing learning rate of some sort. We do note; however, that the step size we use, which is $\frac{1}{n}$ for every point in the trajectory, is different from the step size typically used with Robbins-Monro procedure, which is different at each time step. This means that the sum of our step sizes is finite while the sum of the Robbins-Monro step-sizes diverges. Hence our choice of step size does not give the guarantees typically associated with stochastic optimisation. We use the step sequence since it serves as a useful intermediate stage between simply taking one PG step of equation (11) and using a finite step-sizes, which would mean that the covariance would converge either to zero or diverge to infinity.