Entropic Regularization of Markov Decision Processes

TL;DR

This paper introduces a unified entropic regularization framework for Markov decision processes using $f$-divergences, enabling stable policy learning and providing insights into different divergence choices' effects on reinforcement learning.

Contribution

It generalizes KL-based regularization to a broader family of $f$-divergences, offering closed-form policy updates and a unified actor-critic perspective.

Findings

Closed-form policy improvement for various divergences.

Connection between least-squares value estimation and Pearson $ ext{chi}^2$-divergence.

Asymptotic analysis of divergence effects on standard RL problems.

Abstract

An optimal feedback controller for a given Markov decision process (MDP) can in principle be synthesized by value or policy iteration. However, if the system dynamics and the reward function are unknown, a learning agent must discover an optimal controller via direct interaction with the environment. Such interactive data gathering commonly leads to divergence towards dangerous or uninformative regions of the state space unless additional regularization measures are taken. Prior works proposed bounding the information loss measured by the Kullback-Leibler (KL) divergence at every policy improvement step to eliminate instability in the learning dynamics. In this paper, we consider a broader family of $f$-divergences, and more concretely $\alpha$-divergences, which inherit the beneficial property of providing the policy improvement step in closed form at the same time yielding a corresponding dual objective for policy evaluation. Such entropic proximal policy optimization view gives a unified perspective on compatible actor-critic architectures. In particular, common least-squares value function estimation coupled with advantage-weighted maximum likelihood policy improvement is shown to correspond to the Pearson $\chi^2$-divergence penalty. Other actor-critic pairs arise for various choices of the penalty-generating function $f$. On a concrete instantiation of our framework with the $\alpha$-divergence, we carry out asymptotic analysis of the solutions for different values of $\alpha$ and demonstrate the effects of the divergence function choice on common standard reinforcement learning problems.

Tables

Table 1. Table 1: Empirical policy evaluation and policy improvement objectives for α ∈ { 1 , 2 } 𝛼 1 2 \alpha\in\{1,2\} .

KL Divergence ($\alpha =1$)	Pearson ${\chi }^2$-Divergence ($\alpha =2$)
${\hat{g}}_1\left(w\right)=\eta \log \left({\hat{E}}_t\left[\exp \left(\frac{{\hat{A}}^w(s_t,a_t)}{\eta }\right)\right]\right)$	${\hat{g}}_2\left(w\right)=\frac{1}{2\eta }{\hat{E}}_t\left[{\left({\hat{A}}^w(s_t,a_t)-{\hat{E}}_t\left[{\hat{A}}^w\right]\right)}^2\right]$
${\hat{L}}_1\left(\theta \right)={\hat{E}}_t\left[\log {\pi }_{\theta }\left(a_t|s_t\right)\exp \left(\frac{{\hat{A}}^w(s_t,a_t)-{\hat{g}}_1\left(w\right)}{\eta }\right)\right]$	${\hat{L}}_2\left(\theta \right)=\frac{1}{\eta }{\hat{E}}_t\left[\log {\pi }_{\theta }\left(a_t|s_t\right)\left({\hat{A}}^w(s_t,a_t)-{\hat{E}}_t\left[{\hat{A}}^w\right]+\eta \right)\right]$

Table 2. Table 2: Function f α subscript 𝑓 𝛼 f_{\alpha} , its convex conjugate f α ∗ subscript superscript 𝑓 𝛼 f^{*}_{\alpha} , and their derivatives for some values of α 𝛼 \alpha .

Divergence	$𝜶$	$𝒇\left(𝒙\right)$	${𝒇}^{\prime }\left(𝒙\right)$	${\left({𝒇}^{\ast }\right)}^{\prime }\left(𝒚\right)$	${𝒇}^{\ast }\left(𝒚\right)$	$\mathrm{𝐝𝐨𝐦}{𝒇}^{\ast }$
KL	$1$	$x\log x-\left(x-1\right)$	$\log x$	$e^y$	$e^y-1$	$ℝ$
Reverse KL	$0$	$-\log x+\left(x-1\right)$	$-\frac{1}{x}+1$	$\frac{1}{1-y}$	$-\log \left(1-y\right)$
Pearson ${\chi }^2$	$2$	$\frac{1}{2}{\left(x-1\right)}^2$	$x-1$	$y+1$	$\frac{1}{2}{\left(y+1\right)}^2-\frac{1}{2}$	$y>-1$
Neyman ${\chi }^2$	$-1$	$\frac{{\left(x-1\right)}^2}{2x}$	$-\frac{1}{2x^2}+\frac{1}{2}$	$\frac{1}{\sqrt{1-2y}}$	$-\sqrt{1-2y}+1$
Hellinger	$\frac{1}{2}$	$2{\left(\sqrt{x}-1\right)}^2$	$2-\frac{2}{\sqrt{x}}$	$\frac{4}{{\left(2-y\right)}^2}$	$\frac{2y}{2-y}$

Table 3. Table 3: Chain environment.

Parameter	Value
Number of states	8
Action success probability	0.9
Small and large rewards	(2.0, 10.0)
Number of runs	10
Number of iterations	30
Number of samples	800
Temperature parameters $({\eta }_0,a)$	(15.0, 0.9)

Table 4. Table 4: CliffWalking environment.

Parameter	Value
Punishment for falling from the cliff	$-10.0$
Reward for reaching the goal	100
Number of runs	10
Number of iterations	40
Number of samples	1500
Temperature parameters $({\eta }_0,a)$	(50.0, 0.9)

Table 5. Table 5: FrozenLake environment.

Parameter	Value
Action success probability	0.8
Number of runs	10
Number of iterations	50
Number of samples	2000
Temperature parameters $({\eta }_0,a)$	(1.0, 0.8)

Equations

$$\hat{g}={\hat{E}}_t\left[{\nabla }_{\theta }\log {\pi }_{\theta }{\hat{A}}_t^w\right]$$

$$w=\arg \underset{\tilde{w}}{\min }{\hat{E}}_t\left[\Vert V^{\tilde{w}}(s_t)-{\hat{V}}_t{\Vert }^2\right].$$

$$\theta =\arg \underset{\tilde{\theta }}{\max }{\hat{E}}_t\left[\log {\pi }_{\tilde{\theta }}{\hat{A}}_t^w\right].$$

$$D_f(p\Vert q)=E_q\left[f\left(\frac{p}{q}\right)\right]$$

$$f_{\alpha }(x)=\frac{(x^{\alpha }-1)-\alpha (x-1)}{\alpha (\alpha -1)},\alpha \in \mathbb{R}.$$

$${\nabla }_{\theta }J({\pi }_{\theta })=E_{{\rho }_{{\pi }_{\theta }}(s,a)}[{\nabla }_{\theta }\log {\pi }_{\theta }(a|s)R(s,a)]$$

$${\nabla }_{\theta }\hat{J}={\hat{E}}_t[{\nabla }_{\theta }\log {\pi }_{\theta }(a_t|s_t)R(s_t,a_t)],$$

$${\hat{E}}_t[\log {\pi }_{\theta }(a_t|s_t)R(s_t,a_t)]$$

$$J_{\eta }(\pi )=E_{{\rho }_{\pi }}[R]-\eta D_1({\rho }_{\pi }\Vert {\rho }_{{\pi }_0})$$

$$\underset{\pi }{\mathrm{maximize}}$$

$${\int }_A{\rho }_{\pi }(s^{\prime },a^{\prime })d{a}^{\prime }={\int }_{S\times A}{\rho }_{\pi }(s,a)p(s^{\prime }|s,a)dsda,\forall s^{\prime }\in S,$$

$${\int }_{S\times A}{\rho }_{\pi }(s,a)dsda=1,$$

$${\rho }_{\pi }(s,a)\ge 0,\forall (s,a)\in S\times A.$$

$${\rho }_{\pi }(s,a)={\rho }_{{\pi }_0}(s,a)f_{\ast }^{\prime }\left(\frac{R(s,a)+{\int }_{S}V(s^{\prime })p(s^{\prime }|s,a)d{s}^{\prime }-V(s)-\lambda +\kappa (s,a)}{\eta }\right).$$

$$\underset{V,\lambda ,\kappa }{\mathrm{minimize}}$$

$$\kappa (s,a)\ge 0,\forall (s,a)\in S\times A,$$

$$\arg f_{\ast }\in {\mathrm{range}}_{x\ge 0}{f}^{\prime }(x),\forall (s,a)\in S\times A.$$

$$f_{\alpha }^{\ast }(y)=\frac{1}{\alpha }(1+(\alpha -1)y{)}^{\frac{\alpha }{\alpha -1}}-\frac{1}{\alpha },\text{for}y(1-\alpha )<1.$$

$$\theta =\arg \underset{\tilde{\theta }}{\max }{\hat{E}}_t\left[\log {\pi }_{\tilde{\theta }}(a_t|s_t)f_{\ast }^{\prime }\left(\frac{{\hat{A}}^w(s_t,a_t)-\lambda +\kappa (s_t,a_t)}{\eta }\right)\right].$$

$$\begin{array}[]{llrl}(\textrm{step 1: policy evaluation})&&\underset{w}{\textrm{minimize}}&\hat{g}_{\alpha}(w)\\ (\textrm{step 2: policy improvement})&&\underset{\theta}{\textrm{maximize}}&\hat{L}_{\alpha}(\theta)\end{array}$$

$${\hat{g}}_2(w)\propto {\hat{E}}_t\left[{\left({\hat{A}}^w(s_t,a_t)\right)}^2\right]$$

$${\hat{L}}_2(\theta )\propto {\hat{E}}_t\left[\log {\pi }_{\theta }(a_t|s_t){\hat{A}}^w(s_t,a_t)\right]$$

$$f_{\ast }\left(\frac{A^w(s,a)-\lambda }{\eta }\right)=f_{\ast }(0)+\frac{A^w(s,a)-\lambda }{\eta }{f}_{\ast }^{\prime }(0)+\frac{1}{2}{\left(\frac{A^w(s,a)-\lambda }{\eta }\right)}^2{f}_{\ast }^{\prime \prime }(0)+o\left(\frac{1}{{\eta }^2}\right).$$

$$D_f(\pi \Vert q)=\sum _{a\in \mathcal{A}}q(a)f\left(\frac{\pi (a)}{q(a)}\right),$$

$$f_{\alpha }(x)=\frac{(x^{\alpha }-1)-\alpha (x-1)}{\alpha (\alpha -1)}.$$

$$f_{\alpha }^{\ast }(y)$$

$$(f_{\alpha }^{\ast }{)}^{\prime }(y)$$

$$f^{\ast }(y)+f(x^{\star }(y))=⟨y,x^{\star }(y)⟩,$$

Full text

\pubvolume

21 \issuenum7 \articlenumber674

\historyReceived: 14 June 2019; Accepted: 8 July 2019; Published: 10 July 2019

\TitleEntropic Regularization of Markov Decision Processes

\AuthorBoris Belousov 1,∗\orcidicon and Jan Peters 1,2\orcidicon \AuthorNamesBoris Belousov and Jan Peters

\corresCorrespondence: belousov@ias.tu-darmstadt.de; Tel.: +49-6151-16-25387

1 Introduction

Sequential decision-making problems under uncertainty are described by the mathematical framework of Markov decision processes (MDPs) Puterman (1994). The core problem in MDPs is to find an optimal policy—a mapping from states to actions which maximizes the expected cumulative reward collected by an agent over its lifetime. In reinforcement learning (RL), the agent is additionally assumed to have no prior knowledge about the environment dynamics and the reward function Sutton and Barto (1998). Therefore, direct policy optimization in the RL setting can be seen as a form of stochastic black-box optimization: the agent proposes a query point in the form of a policy, the environment evaluates this point by computing the expected return, after that the agent updates the proposal and the process repeats Deisenroth et al. (2013). There are two conceptual steps in this scheme known as policy evaluation and policy improvement Bellman (1957). Both steps require function approximation in high-dimensional and continuous state-action spaces due to the curse of dimensionality Bellman (1957). Therefore, statistical learning approaches are employed to approximate the value function of a policy and to perform policy improvement based on the data collected from the environment.

In contrast to traditional supervised learning, in reinforcement learning, the data distribution changes with every policy update. State-of-the-art generalized policy iteration algorithms Kakade (2001); Peters et al. (2010); Schulman et al. (2015, 2017) are mindful of this covariate shift problem Shimodaira (2000), taking active measures to account for it. To smoothen the learning dynamics, these algorithms limit the information loss between successive policy updates as measured by the KL divergence or approximations thereof Neu et al. (2017). In the optimization literature, such approaches are categorized as proximal (or trust region) algorithms Parikh (2014).

The choice of the divergence function determines the geometry of the information manifold Nielsen (2018). Recently, in particular in the area of implicit generative modeling Goodfellow et al. (2014), the choice of the divergence function was shown to have a dramatic effect both on the optimization performance Bottou et al. (2017) and the perceptual quality of the generated data when various $f$-divergences were employed Nowozin et al. (2016). In this paper, we carry over the idea of using generalized entropic proximal mappings Teboulle (1992) given by an $f$-divergence to reinforcement learning. We show that relative entropy policy search Peters et al. (2010), framed as an instance of stochastic mirror descent Nemirovski and Yudin (1983); Beck and Teboulle (2003) as suggested by Neu et al. (2017), can be extended to use any divergence measure from the family of $f$-divergences. The resulting algorithm provides insights into the compatibility of policy and value function update rules in actor-critic architectures, which we exemplify on several instantiations of the generic $f$-divergence with representatives from the parametric family of $\alpha$-divergences Chernoff (1952); Amari (1985); Cichocki and Amari (2010).

2 Background

This section provides the necessary background on policy gradients Deisenroth et al. (2013) and entropic penalties Teboulle (1992) for later derivations and analysis. Standard RL notation Thomas and Okal (2015) is used throughout.

2.1 Policy Gradient Methods

Policy search algorithms Deisenroth et al. (2013) commonly use the gradient estimator of the following form Sutton et al. (1999)

[TABLE]

where $\pi_{\theta}(a|s)$ is a stochastic policy and $\hat{A}^{w}_{t}(s_{t},a_{t})$ is an estimator of the advantage function at timestep $t$. Expectation $\hat{E}_{t}[\dots]$ indicates an empirical average over a finite batch of samples, in an algorithm that alternates between sampling and optimization. The advantage estimate $\hat{A}^{w}_{t}$ in (1) can be obtained from an estimate of the value function Peters and Schaal (2008); Schulman et al. (2016), which in its turn is found by least-squares estimation. Specifically, if $V^{w}(s)$ denotes a parametric value function, and if $\hat{V}_{t}=\sum_{k=0}^{\infty}\gamma^{k}R_{t+k}$ is taken as its rollout-based estimate, then the parameters $w$ can be found as

[TABLE]

The advantage estimate $\hat{A}_{t}^{w}=\sum_{k=0}^{\infty}\gamma^{k}\delta^{w}_{t+k}$ is then obtained by summing the temporal difference errors $\delta^{w}_{t}=R_{t}+\gamma V^{w}(s_{t+1})-V^{w}(s_{t})$, also known as the Bellman residuals. Treating $\hat{A}^{w}_{t}$ as fixed for the purpose of policy improvement, we can view (1) as the gradient of an advantage-weighted log-likelihood; therefore, the policy parameters $\theta$ can be found as

[TABLE]

Thus, actor-critic algorithms that use the gradient estimator (1) to update the policy can be viewed as instances of the generalized policy iteration scheme, alternating between policy evaluation (2) and policy improvement (3). In the following, we will see that the actor-critic pair (2) and (3), that combines least-squares value function fitting with linear-in-the-advantage-weighted maximum likelihood policy improvement, is just one representative from a family of such actor-critic pairs arising for different choices of the $f$-divergence penalty within our entropic proximal policy optimization framework.

2.2 Entropic Penalties

The term entropic penalties Teboulle (1992) refers to both $f$-divergences and Bregman divergences. In this paper, we will focus on $f$-divergences, leaving generalization to Bregman divergences for future work. The $f$-divergence Csiszár (1963) between two distributions $P$ and $Q$ with densities $p$ and $q$ is defined as

[TABLE]

where $f$ is a convex function on $(0,\infty)$ with $f(1)=0$ and $P$ is assumed to be absolutely continuous with respect to $Q$. For example, the KL divergence corresponds to $f_{1}(x)=x\log x-(x-1)$, with the formula also applicable to unnormalized distributions Zhu and Rohwer (1995). Many common divergences lie on the curve of $\alpha$-divergences Chernoff (1952); Amari (1985) defined by a special choice of the generator function Cichocki and Amari (2010)

[TABLE]

The $\alpha$-divergence $D_{\alpha}=D_{f_{\alpha}}$ will be used as the primary example of the $f$-divergence throughout the paper. For more details on the $\alpha$-divergence and its properties, see Appendix A. Noteworthy is the symmetry of the $\alpha$-divergence with respect to $\alpha=0.5$, which relates reverse divergences as $D_{0.5+\beta}(p\|q)=D_{0.5-\beta}(q\|p)$.

3 Entropic Proximal Policy Optimization

Consider the average-reward RL setting Sutton and Barto (1998), where the dynamics of an ergodic MDP are given by the transition density $p(s^{\prime}|s,a)$. An intelligent agent can modulate the system dynamics by sampling actions $a$ from a stochastic policy $\pi(a|s)$ at every time step of the evolution of the dynamical system. The resulting modulated Markov chain with transition kernel $p_{\pi}(s^{\prime}|s)=\int_{A}p(s^{\prime}|s,a)\pi(a|s)da$ converges to a stationary state distribution $\mu_{\pi}(s)$ as time goes to infinity. This stationary state distribution induces a state-action distribution $\rho_{\pi}(s,a)=\mu_{\pi}(s)\pi(a|s)$, which corresponds to visitation frequencies of state-action pairs Puterman (1994). The goal of the agent is to steer the system dynamics to desirable states. Such objective is commonly encoded by the expectation of a random variable $R\colon S\times A\to\mathbb{R}$ called reward in this context. Thus, the agent seeks a policy that maximizes the expected reward $J(\pi)=E_{\rho_{\pi}(s,a)}[R(s,a)]$.

In reinforcement learning, neither the reward function $R$ nor the system dynamics $p(s^{\prime}|s,a)$ are assumed to be known. Therefore, to maximize (or even evaluate) the objective $J(\pi)$, the agent must sample a batch of experiences in the form of tuples $(s,a,r,s^{\prime})$ from the dynamics and use an empirical estimate $\hat{J}=\hat{E}_{t}[R(s_{t},a_{t})]$ as a surrogate for the original objective. Since the gradient of the expected reward with respect to the policy parameters can be written as Williams (1992)

[TABLE]

with a corresponding sample-based counterpart

[TABLE]

one may be tempted to optimize a sample-based objective

[TABLE]

on a fixed batch of data $\{(s,a,r,s^{\prime})_{t}\}_{t=1}^{N}$ till convergence. However, such an approach ignores the fact that sampling distribution $\rho_{\pi_{\theta}}(s,a)$ itself depends on the policy parameters $\theta$; therefore, such greedy optimization aims at a wrong objective Peters et al. (2010). To have the correct objective, the dataset must be sampled anew after every parameter update—doing otherwise will lead to overfitting and divergence. This problem is known in statistics as the covariate shift problem Shimodaira (2000).

3.1 Fighting Covariate Shift via Trust Regions

A principled way to account for the change in the sampling distribution at every policy update step is to construct an auxiliary local objective function that can be safely optimized till convergence. Relative entropy policy search (REPS) algorithm Peters et al. (2010) proposes a candidate for such an objective

[TABLE]

with $\pi_{0}$ being the current policy under which the data samples were collected, policy $\pi$ being the improvement policy that needs to be found, and $\eta>0$ being a ‘temperature’ parameter that determines how much the next policy can deviate from the current one. The original formulation employs a relative entropy trust region constraint $D_{1}$ with radius $\varepsilon$ instead of a penalty, which allows for finding the optimal temperature $\eta$ as a function of the trust region radius $\varepsilon$.

Importantly, the objective function (5) can be optimized in closed form for policy $\pi$ (i.e., treating the policy itself as a variable and not its parameters, in contrast to standard policy gradients). To that end, several constraints on $\rho_{\pi}$ are added to ensure stationarity with respect to the given MDP Peters et al. (2010). In a similar vein, we can solve Problem (5) with respect to $\pi$ for any $f$-divergence with a twice differentiable generator function $f$.

3.2 Policy Optimization with Entropic Penalties

Following the intuition of REPS, we introduce an $f$-divergence penalized optimization problem that the learning agent must solve at every policy iteration step

[TABLE]

The agent seeks a policy that maximizes the expected reward and does not deviate from the current policy too much. The first constraint in (6) ensures that the policy is compatible with the system dynamics, and the latter two constraints ensure that $\pi$ is a proper probability distribution. Please note that $\pi$ enters Problem (6) indirectly through $\rho_{\pi}$. Since the objective has the form of free energy Wainwright and Jordan (2007) in $\rho_{\pi}$ with an $f$-divergence playing the role of the usual KL, the solution can be expressed through the derivative of the convex conjugate function $f_{*}^{\prime}$, as shown for general nonlinear problems in Teboulle (1992),

[TABLE]

Here, $\{V(s),\lambda,\kappa(s,a)\}$ are the Lagrange dual variables corresponding to the three constraints in (6), respectively. Although we get a closed-form solution for $\rho_{\pi}$, we still need to solve the dual optimization problem to get the optimal dual variables

[TABLE]

Remarkably, the advantage function $A^{V}(s,a)=R(s,a)+\int_{S}V(s^{\prime})p(s^{\prime}|s,a)ds^{\prime}-V(s)$ emerges automatically in the dual objective. The advantage function also appears in the penalty-free linear programming formulation of policy improvement Puterman (1994), which corresponds to the zero-temperature limit $\eta\to 0$ of our formulation. Thanks to the fact that the dual objective in (8) is given as an expectation with respect to $\rho_{\pi_{0}}$, it can be straightforwardly estimated from rollouts. The last constraint in (8) on the argument of $f_{*}$ is easy to evaluate for common $\alpha$-divergences. Indeed, the convex conjugate $f_{\alpha}^{*}$ of the generator function (4) is given by

[TABLE]

Thus, the constraint on $\arg f_{*}$ in (4) is just a linear inequality $y(1-\alpha)<1$ for any $\alpha$-divergence.

3.3 Value Function Approximation

For small grid-world problems, one can solve Problem (8) exactly for $V(s)$. However, for larger problems or if the state space is continuous, one must resort to function approximation. Assume we plug an expressive function approximator $V^{w}(s)$ in (8), then vector $w$ becomes a new vector of parameters in the dual objective. Later, it will be shown that minimizing the dual when $\eta\to\infty$ is closely related to minimizing the mean squared Bellman error.

3.4 Sample-Based Algorithm for Dual Optimization

To solve Problem (8) in practice, we gather a batch of samples from policy $\pi_{0}$ and replace the expectation in the objective with a sample average. Please note that in principle one also needs to estimate the expectation of the future rewards $\int_{S}V(s^{\prime})p(s^{\prime}|s,a)ds^{\prime}$. However, since the probability of visiting the same state-action pair in continuous space is zero, one commonly estimates this integral from a single sample Deisenroth et al. (2013), which is equivalent to assuming deterministic system dynamics. Inequality constraints in (8) are linear and they must be imposed for every $(s,a)$ pair in the dataset.

3.5 Parametric Policy Fitting

Assume Problem (8) is solved on a current batch of data sampled from $\pi_{0}$ and thus the optimal dual variables $\{V(s),\lambda,\kappa(s,a)\}$ are given. Equation (7) allows one to evaluate the new density $\rho_{\pi}(s,a)$ on any pair $(s,a)$ from the dataset. However, it does not yield the new policy $\pi$ directly because representation (7) is variational. A common approach Deisenroth et al. (2013) is to assume that the policy is represented by a parameterized conditional density $\pi_{\theta}(a|s)$ and fit this density to the data using maximum likelihood.

To fit a parametric density $\pi_{\theta}(a|s)$ to the true solution $\pi(a|s)$ given by (7), we minimize the KL divergence $D_{1}(\rho_{\pi}\|\rho_{\pi_{\theta}})$. Minimization of this KL is equivalent to maximization of the weighted maximum likelihood $\hat{E}[f_{*}^{\prime}(\dots)\log\rho_{\pi_{\theta}}]$. Unfortunately, distribution $\rho_{\pi_{\theta}}(s,a)=\mu_{\pi_{\theta}}(s)\pi_{\theta}(a|s)$ is in general not known because $\mu_{\pi_{\theta}}(s)$ does not only depend on the policy but also on the system dynamics. Assuming the effect of policy parameters on the stationary state distribution is small Deisenroth et al. (2013), we arrive at the following optimization problem for fitting the policy parameters

[TABLE]

Compare our policy improvement step (10) to the commonly used advantage-weighted maximum likelihood (ML) objective (3). They look surprisingly similar (especially if $f_{*}^{\prime}(y)=y$ is a linear function), which is not a coincidence and will be systematically explained in the next sections.

3.6 Temperature Scheduling

The ‘temperature’ parameter $\eta$ trades off reward vs divergence, as can be seen in the objective function in Problem (6). In practice, devising a schedule for $\eta$ may be hard because $\eta$ is sensitive to reward scaling and policy parameterization. A more intuitive way to impose the $f$-divergence proximity condition is by adding it as a constraint $D_{f}(\rho_{\pi}\|\rho_{\pi_{0}})\leq\varepsilon$ with a fixed $\varepsilon$ and then treating the temperature $\eta\geq 0$ as an optimization variable. Such formulation is easy to incorporate into the dual (8) by adding a term $\eta\varepsilon$ to the objective and a constraint $\eta\geq 0$ to the list of constraints. Constraint-based formulation was successfully used before with a KL divergence constraint Peters et al. (2010) and with its quadratic approximation Kakade (2001); Schulman et al. (2015).

3.7 Practical Algorithm for Continuous State-Action Spaces

Our proposed approach for entropic proximal policy optimization is summarized in Algorithm 1. Following the generalized policy iteration scheme, we (i) collect data under a given policy, (ii) evaluate the policy by solving (8), and (iii) improve the policy by solving (10). In the following section, several instantiations of Algorithm 1 with different choices of function $f$ will be presented and studied.

4 High- and Low-Temperature Limits; $\alpha$-Divergences;

Analytic Solutions and Asymptotics

How does the $f$-divergence penalty influence policy optimization? How should one choose the generator function $f$? What role does the step size play in optimization? This section will try to answer these and related questions. First, two special choices of the penalty function $f$ are presented, which reveal that the common practice of using mean squared Bellman error minimization coupled with advantage reweighted policy update is equivalent to imposing a Pearson $\chi^{2}$-divergence penalty. Second, high- and low-temperature limits are studied, on one hand revealing the special role the Pearson $\chi^{2}$-divergence plays, being the high-temperature limit of all smooth $f$-divergences, and on the other hand establishing a link to the linear programming formulation of policy search as the low-temperature limit of our entropic penalty-based framework.

4.1 KL Divergence ($\alpha=1$)

and Pearson $\chi^{2}$-Divergence ($\alpha=2$)

As can be deduced from the form of (10), great simplifications occur when $f_{*}^{\prime}(y)$ is a linear function ($\alpha=2$, see (9)) or an exponential function ($\alpha=1$). The fundamental reason for such simplifications lies in the fact that linear and exponential functions are homomorphisms with respect to addition. This allows, in particular, discovery of a closed-form solution for the dual variable $\lambda$ and thus eliminate it from the optimization. Moreover, in these two special cases, the dual variables $\kappa(s,a)$ can also be eliminated. They are responsible for non-negativity of probabilities: when $\alpha=1$ (KL), $\kappa(s,a)=0$ uniformly for all $\eta\geq 0$, when $\alpha=2$ (Pearson), $\kappa(s,a)=0$ for sufficiently big $\eta$. Table 1 gives the corresponding empirical actor-critic optimization objective pairs. A generic primal-dual actor-critic algorithm with an $\alpha$-divergence penalty performs two steps

[TABLE]

inside a policy iteration loop. It is worth comparing the explicit formulas in Table 1 to the customarily used objectives (2) and (3). To make the comparison fair, notice that (2) and (3) correspond to discounted infinite horizon formulation with discount factor $\gamma\in(0,1)$ whereas formulas in Table 1 are derived for the average-reward setting. In general, the difference between these two settings can be ascribed to an additional baseline that must be subtracted in the average reward setting Sutton and Barto (1998). In our derivations, the baseline corresponds to the dual variable $\lambda$, as in classical linear programming formulation of policy iteration Puterman (1994), and it is automatically gets subtracted from the advantage (see (8)).

Mean Squared Error Minimization with Advantage Reweighting is Equivalent to Pearson Penalty

The baseline for $\alpha=2$ is given by the average advantage $\lambda_{2}=\hat{E}_{t}\left[\hat{A}^{w}(s_{t},a_{t})\right]$, which also equals the average return in our setting Sutton and Barto (1998); Puterman (1994). Therefore, to translate the formulas from Table 1 to the discounted infinite horizon form (2) and (3), we need to remove the baseline and add discounting to the advantage; that is, set $A^{w}(s,a)=R(s,a)+\gamma\int_{S}V^{w}(s^{\prime})p(s^{\prime}|s,a)ds^{\prime}-V^{w}(s)$. Then the dual objective

[TABLE]

is proportional to the average squared advantage. Naive optimization of (11) leads to the family of residual gradient algorithms Baird (1995); Dann et al. (2014). However, if the same Monte Carlo estimate of the value function is used as in (2), then (11) and (2) are exactly equivalent. The same holds for the Pearson actor

[TABLE]

and the standard policy improvement (3) provided that $\eta=\hat{E}_{t}\left[\hat{A}^{w}(s_{t},a_{t})\right]$. That means (12) is equivalent to (3) if the weight of the divergence penalty is equal to the expected return.

4.2 High- and Low-Temperature Limits

In the previous subsection, we established a direct correspondence between the least-squares value function fitting coupled with the advantage-weighted maximum likelihood policy parameters estimation (2) and (3) and the dual-primal pair of optimization problems (11) and (12) arising from our Algorithm 1 for the special choice of the Pearson $\chi^{2}$-divergence penalty. In this subsection, we will show that this is not a coincidence but a manifestation of the fundamental fact that the Pearson $\chi^{2}$-divergence is the quadratic approximation of any smooth $f$-divergence about unity.

4.2.1 High Temperatures: All Smooth $f$-Divergences Tend Towards Pearson $\chi^{2}$-Divergence

There are two ways to show the independence of the primal-dual solution (8)–(10) on the choice of the divergence penalty: either exactly solve an approximate problem or approximate the exact solution of the original problem. In the first case, the penalty is replaced with its Taylor expansion at $\eta\to\infty$, which turns out to be the Pearson $\chi^{2}$-divergence, and then the derivation becomes equivalent to the natural policy gradient derivation Kakade (2001). In the second case, the exact solution (8)–(10) is expanded by Taylor: for big $\eta$, dual variables $\kappa(s,a)$ can be dropped if $\rho_{\pi_{0}}(s,a)>0$, which yields

[TABLE]

By definition of the $f$-divergence, the generator function $f$ satisfies the condition $f(1)=0$. Without loss of generality Sason and Verdu (2016), one can impose an additional constraint $f^{\prime}(1)=0$ for convenience. Such constraint ensures that the graph of the function $f(x)$ lies entirely in the upper half-plane, touching the $x$-axis at a single point $x=1$. From the definition of the convex conjugate $f_{*}^{\prime}=(f^{\prime})^{-1}$, we can deduce that $f_{*}^{\prime}(0)=1$ and $f_{*}(0)=0$; by rescaling, it is moreover possible to set $f^{\prime\prime}(1)=f_{*}^{\prime\prime}(0)=1$. These properties are automatically satisfied by the $\alpha$-divergence, which can be verified by a direct computation. With this in mind, it is straightforward to see that substitution of (13) into (8) yields precisely the quadratic objective $\hat{g}_{2}(w)$ from Table 1, the difference being of the second order in $1/\eta$.

To obtain the asymptotic policy update objective, one can expand (10) in the high-temperature limit $\eta\to\infty$ and observe that it equals $\hat{L}_{2}(\theta)$ from Table 1 with the difference being of the second order in $1/\eta$. Therefore, it is established that the choice of the divergence function plays a minor role for big temperatures (small policy update steps). Since this is the mode in which the majority of iterative algorithms operate, our entropic proximal policy optimization point of view provides a rigorous justification for the common practice of using the mean squared Bellman error objective for value function fitting and the advantage-weighted maximum likelihood objective for policy improvement.

4.2.2 Low Temperatures: Linear Programming Formulation Emerges in the Limit

Setting $\eta$ to a small number is equivalent to allowing large policy update steps because $\eta$ is the weight of the divergence penalty in the objective function (6). Such regime is rather undesirable in reinforcement learning because of the covariate shift problem mentioned in the introduction. Problem (6) for $\eta\to 0$ turns into a well-studied linear programming formulation Puterman (1994); Neu et al. (2017) that can be readily applied if the model $\{p(s^{\prime}|s,a),R(s,a)\}$ is known.

It is not straightforward to derive the asymptotics of policy evaluation (8) and policy improvement (10) for a general smooth $f$-divergence in the low-temperature limit $\eta\to 0$ because the dual variables $\kappa(s,a)$ do not disappear, in contrast to the high-temperature limit (13). However, for the KL divergence penalty (see Table 1), one can show that the policy evaluation objective $g_{1}(w)$ tends towards the supremum of the advantage $g_{1}(w)\to\sup_{s,a}A^{w}(s,a)$; the optimal policy is deterministic, $\pi(a|s)\to\delta(a-\arg\sup_{b}A^{w}(s,b))$, therefore $L(\theta)\to\log\pi_{\theta}(\bar{a}|\bar{s})$ with $(\bar{s},\bar{a})=\arg\sup_{s^{\prime},a^{\prime}}A^{w}(s^{\prime},a^{\prime})$.

5 Empirical Evaluations

To develop an intuition regarding the influence of the entropic penalties on policy improvement, we first consider a simplified version of the reinforcement learning problem—namely the stochastic multi-armed bandit problem (Bubeck and Cesa-Bianchi, 2012). In this setting, our algorithm is closely related to the family of Exp3 algorithms (Auer et al., 2003), originally motivated by the adversarial bandit problem. Subsequently, we evaluate our approach in the standard reinforcement learning setting.

5.1 Illustrative Experiments on Stochastic Multi-Armed Bandit Problems

In the stochastic multi-armed bandit problem (Bubeck and Cesa-Bianchi, 2012), at every time step $t\in\{1,\dots,T\}$, an agent chooses among $K$ actions $a\in\mathcal{A}$. After every choice $a_{t}=a$, it receives a noisy reward $R_{t}=R(a_{t})$ drawn from a distribution with mean $Q(a)$. The goal of the agent is to maximize the expected total reward $J=E[\sum_{t=1}^{T}R_{t}]$. Given the true values $Q(a)$, the optimal strategy is to always choose the best action, $a^{*}_{t}=\operatorname*{arg\,max}_{a}{Q(a)}$. However, due to the lack of knowledge, the agent faces the exploration-exploitation dilemma. A generic way to encode the exploration-exploitation trade-off is by introducing a policy $\pi_{t}$, i.e., a distribution from which the agent draws actions $a_{t}\sim\pi_{t}$. Thus, the question becomes: given the current policy $\pi_{t}$ and the current estimate of action values $\hat{Q}_{t}$, what should the policy $\pi_{t+1}$ at the next time step be? Unlike the choice of the best action under perfect information, such sampling policies are hard to derive from first principles (Ghavamzadeh et al., 2015).

We apply our generic Algorithm 1 to the stochastic multi-armed bandit problem to illustrate the effects of the divergence choice. The value function disappears because there is no state and no system dynamics in this problem. Therefore, the estimate $\hat{Q}_{t}$ plays the role of the advantage, and the dual optimization (8) is performed only with respect to the remaining Lagrange multipliers.

5.1.1 Effects of $\alpha$ on Policy Improvement

Figure 1 shows the effects of the $\alpha$-divergence choice on policy updates. We consider a $10$-armed bandit problem with arm values $Q(a)\sim\mathcal{N}(0,1)$ and keep the temperature fixed at $\eta=2$ for all values of $\alpha$. Several iterations starting from an initial uniform policy are shown in the figure for comparison. Extremely large positive and negative values of $\alpha$ result in $\varepsilon$-elimination and $\varepsilon$-greedy policies, respectively. Small values of $\alpha$, in contrast, weigh actions according to their values. Policies for $\alpha<1$ are peaked and heavy-tailed, eventually turning into $\varepsilon$-greedy policies when $\alpha\to-\infty$. Policies for $\alpha\geq 1$ are more uniform, but they put zero mass on bad actions, eventually turning into $\varepsilon$-elimination policies when $\alpha\to\infty$. For $\alpha\geq 1$, policy iteration may spend a lot of time in the end deciding between two best actions, whereas for $\alpha<1$ the final convergence is faster.

5.1.2 Effects of $\alpha$ on Regret

The average regret $C_{n}=nQ_{\max}-\mathbb{E}[\sum_{t=0}^{n-1}R_{t}]$ is shown in Figure 2 for different values of $\alpha$ as a function of the time step $n$ with $95\%$ confidence error bars. The performance of the UCB algorithm (Bubeck and Cesa-Bianchi, 2012) is also shown for comparison. The presented results are obtained in a $20$-armed bandit environment where rewards have Gaussian distribution $R(a)\sim\mathcal{N}(Q(a),0.5)$. Arm values are estimated from observed rewards and the policy is updated every $20$ time steps. The temperature parameter $\eta$ is decreased starting from $\eta=1$ after every policy update according to the schedule $\eta^{+}=\beta\eta$ with $\beta=0.8$. Results are averaged over $400$ runs. In general, extreme $\alpha$’s accumulate more regret. However, they eventually focus on a single action and flatten out. Small $\alpha$’s accumulate less regret, but they may keep exploring sub-optimal actions longer. Values of $\alpha\in[0,2]$ perform comparably with UCB after around $400$ steps, once reliable estimates of values have been obtained.

Figure 3 shows the average regret after a given number of time steps as a function of the divergence type $\alpha$. As can be seen from the figure, smaller values of $\alpha$ result in lower regret. Large negative $\alpha$’s correspond to $\varepsilon$-greedy policies, which oftentimes prematurely converge to a sub-optimal action, failing to discover the optimal action for a long time if the exploration probability $\varepsilon$ is small. Large positive $\alpha$’s correspond to $\varepsilon$-elimination policies, which may by mistake completely eliminate the best action or spend a lot of time deciding between two options in the end of learning, accumulating more regret. The optimal value of the parameter $\alpha$ depends on the time horizon for which the policy is being optimized. Depending on the horizon, the minimum of the curves shifts from slightly negative $\alpha$’s towards the range $\alpha\in[0,2]$ with increasing time horizon.

5.2 Empirical Evaluations on Ergodic MDPs

We evaluate our policy iteration algorithm with $f$-divergence on standard grid-world reinforcement learning problems from OpenAI Gym (Brockman et al., 2016). The environments that terminate or have absorbing states are restarted during data collection to ensure ergodicity. Figure 4 demonstrates the learning dynamics on different environments for various choices of the divergence function. Parameter settings and other implementation details can be found in Appendix B. In summary, one can either promote risk averse behavior by choosing $\alpha<0$, which may, however, result in sub-optimal exploration, or one can promote risk seeking behavior with $\alpha>1$, which may lead to overly aggressive elimination of options. Our experiments suggest that the optimal balance should be found in the range $\alpha\in[0,1]$. It should be noted that the effect of the $\alpha$-divergence on policy iteration is not linear and not symmetric with respect to $\alpha=0.5$, contrary to what one could have expected given the symmetry of the $\alpha$-divergence as a function of $\alpha$. For example, switching from $\alpha=-3$ to $\alpha=-2$ may have little effect on policy iteration, whereas switching from $\alpha=3$ to $\alpha=4$ may have a much more pronounced influence on the learning dynamics.

6 Related Work

Apart from computational advantages, information-theoretic approaches provide a solid framework for describing and studying aspects of intelligent behavior Tishby and Polani (2011), from autonomy Bertschinger et al. (2008) and curiosity Still and Precup (2012) to bounded rationality Genewein et al. (2015) and game theory Wolpert (2006).

Entropic proximal mappings were introduced in Teboulle (1992) as a general framework for constructing approximation and smoothing schemes for optimization problem. Problem formulation (6) presented here can be considered as an application of this general theory to policy optimization in Markov decision processes. Following the recent work Neu et al. (2017) that establishes links between popular in reinforcement learning KL-divergence-regularized policy iteration algorithms Peters et al. (2010); Schulman et al. (2015) and a well-known in optimization stochastic mirror descent algorithm Nemirovski and Yudin (1983); Beck and Teboulle (2003), one can view our Algorithm 1 as an analog of the mirror descent with an $f$-divergence penalty.

Concurrent works Geist et al. (2019); Li et al. (2019) consider similar regularized formulations, although in the policy space instead of the state-action distribution space and in the infinite horizon discounted setting instead of the average-reward setting. The $\alpha$-divergence in its entropic form, i.e., when the base measure is a uniform distribution, was used in several papers under the name Tsallis entropy Nachum et al. (2018); Lee et al. (2019, 2018a, 2018b), where its sparsifying effect was exploited in large discrete action spaces.

An alternative proximal reinforcement learning scheme was introduced in Mahadevan et al. (2014) based on the extragradient method for solving variational inequalities and leveraging operator splitting techniques. Although the idea of exploiting proximal maps and updates in the primal and dual spaces is similar to ours, regularization in Mahadevan et al. (2014) is applied in the value function space to smoothen generalized TD learning algorithms, whereas we study regularization in the primal space.

7 Conclusions

We presented a framework for deriving actor-critic algorithms as pairs of primal-dual optimization problems resulting from regularization of the standard expected return objective with so-called entropic penalties in the form of an $f$-divergence. Several examples with $\alpha$-divergence penalties have been worked out in detail. In the limit of small policy update steps, all $f$-divergences with twice differentiable generator function $f$ are approximated by the Pearson $\chi^{2}$-divergence, which was shown to yield the most commonly used in reinforcement learning pair of actor-critic updates. Thus, our framework provides a sound justification for the common practice of minimizing mean squared Bellman error in the policy evaluation step and fitting policy parameters by advantage-weighted maximum likelihood in the policy improvement step.

In the future work, incorporating non-differentiable generator functions, such as the absolute value that corresponds to the total variation distance, may provide a principled explanation for the empirical success of the algorithms not accounted for by our current smooth $f$-divergence framework, such as the proximal policy optimization algorithm Schulman et al. (2017). Establishing a tighter connection between online convex optimization that employs Bregman divergences and reinforcement learning will likely yield both a deeper understanding of the optimization dynamics in RL and allow for improved practical algorithms building on the firm fundament of optimization theory.

\authorcontributions

Conceptualization, B.B. and J.P.; investigation, B.B. and J.P.; software, B.B.; supervision, J.P.; writing, B.B. and J.P.

\funding

This project has received funding from the European Union’s Horizon 2020 research and innovation program under grant agreement No. 640554.

Acknowledgements.

We thank Hany Abdulsamad for many insightful discussions. \conflictsofinterest The authors declare no conflict of interest.

\appendixtitlesno

Appendix A

This section provides the background on the $f$-divergence, the $\alpha$-divergence, and the convex conjugate function, highlighting the key properties required for our derivations.

The $f$-divergence (Csiszár, 1963; Morimoto, 1963; Ali and Silvey, 1966) generalizes many similarity measures between probability distributions (Sason and Verdu, 2016). For two distributions $\pi$ and $q$ on a finite set $\mathcal{A}$, the $f$-divergence is defined as

[TABLE]

where $f$ is a convex function on $(0,\infty)$ such that $f(1)=0$. For example, the KL divergence corresponds to $f_{KL}(x)=x\log x$. Please note that $\pi$ must be absolutely continuous with respect to $q$ to avoid division by zero, i.e., $q(a)=0$ implies $\pi(a)=0$ for all $a\in\mathcal{A}$. We additionally assume $f$ to be continuously differentiable, which includes all cases of interest for us. The $f$-divergence can be generalized to unnormalized distributions. For example, the generalized KL divergence (Zhu and Rohwer, 1995) corresponds to $f_{1}(x)=x\log x-(x-1)$. The derivations in this paper benefit from employing unnormalized distributions and subsequently imposing the normalization condition as a constraint.

The $\alpha$-divergence (Chernoff, 1952; Amari, 1985) is a one-parameter family of $f$-divergences generated by the $\alpha$-function $f_{\alpha}(x)$ with $\alpha\in\mathbb{R}$. The particular choice of the family of functions $f_{\alpha}$ is motivated by generalization of the natural logarithm (Cichocki and Amari, 2010). The $\alpha$-logarithm $\log_{\alpha}(x)=(x^{\alpha-1}-1)/(\alpha-1)$ is a power function for $\alpha\neq 1$ that turns into the natural logarithm for $\alpha\to 1$. Replacing the natural logarithm in the derivative of the KL divergence $f_{1}^{\prime}=\log x$ by the $\alpha$-logarithm and integrating $f_{\alpha}^{\prime}$ under the condition that $f_{\alpha}(1)=0$ yields the $\alpha$-function

[TABLE]

The $\alpha$-divergence generalizes the KL divergence, reverse KL divergence, Hellinger distance, Pearson $\chi^{2}$-divergence, and Neyman (reverse Pearson) $\chi^{2}$-divergence. Figure 5 displays well-known $\alpha$-divergences as points on the parabola $y=\alpha(\alpha-1)$. For every divergence, there is a reverse divergence symmetric with respect to the point $\alpha=0.5$, corresponding to the Hellinger distance.

The convex conjugate of $f(x)$ is defined as $f^{*}(y)=\sup_{x\in\operatorname{dom}f}\{\langle y,x\rangle-f(x)\}$, where the angle brackets $\langle y,x\rangle$ denote the dot product (Boyd and Vandenberghe, 2004). The key property $(f^{*})^{\prime}=(f^{\prime})^{-1}$ relating the derivatives of $f^{*}$ and $f$ yields Table 2, which lists common functions $f_{\alpha}$ together with their convex conjugates and derivatives. In the general case (14), the convex conjugate and its derivative are given by

[TABLE]

Function $f_{\alpha}$ is convex, non-negative, and attains minimum at $x=1$ with $f_{\alpha}(1)=0$. Function $(f_{\alpha}^{*})^{\prime}$ is positive on its domain with $(f_{\alpha}^{*})^{\prime}(0)=1$. Function $f_{\alpha}^{*}$ has the property $f_{\alpha}^{*}(0)=0$. The linear inequality constraint (15) on the $\operatorname{dom}f_{\alpha}^{*}$ follows from the requirement $\operatorname{dom}f_{\alpha}=(0,\infty)$. Another result from convex analysis crucial to our derivations is Fenchel’s equality

[TABLE]

where $x^{\star}(y)=\operatorname*{arg\,sup}_{x\in\operatorname{dom}f}\{\langle y,x\rangle-f(x)\}$. We will occasionally put the conjugation symbol at the bottom, especially for the derivative of the conjugate function $f_{*}^{\prime}=(f^{*})^{\prime}$.

Appendix B

In all experiments, the temperature parameter $\eta$ is exponentially decayed $\eta_{i+1}=\eta_{0}a^{i}$ in each iteration $i=0,1,\dots$. The choice of $\eta_{0}$ and $a$ depends on the scale of the rewards and the number of samples collected per policy update. Tables for each environment list these parameters along with the number of samples per policy update, the number of policy iteration steps, and the number of runs for averaging the results. Where applicable, environment-specific settings are also listed. (see the Tables 3–5)

\reftitle

References