Similarities between policy gradient methods (PGM) in Reinforcement learning (RL) and supervised learning (SL)

TL;DR

This paper reveals that policy gradient methods in reinforcement learning can be viewed as supervised learning problems by replacing true labels with discounted rewards, highlighting a fundamental connection between the two learning paradigms.

Contribution

The paper provides a new proof linking policy gradient methods to supervised learning, emphasizing their relationship through reward functions and cross-entropy.

Findings

Policy gradient methods can be reformulated as supervised learning.

Interchanging labels and pseudo rewards affects learning outcomes.

Modifying reward functions can establish further connections with supervised learning.

Abstract

Reinforcement learning (RL) is about sequential decision making and is traditionally opposed to supervised learning (SL) and unsupervised learning (USL). In RL, given the current state, the agent makes a decision that may influence the next state as opposed to SL (and USL) where, the next state remains the same, regardless of the decisions taken, either in batch or online learning. Although this difference is fundamental between SL and RL, there are connections that have been overlooked. In particular, we prove in this paper that gradient policy method can be cast as a supervised learning problem where true label are replaced with discounted rewards. We provide a new proof of policy gradient methods (PGM) that emphasizes the tight link with the cross entropy and supervised learning. We provide a simple experiment where we interchange label and pseudo rewards. We conclude that other relationships with SL could be made if we modify the reward functions wisely.

Tables

Table 1. Table 1: Comparing SL and RL for REINFORCE

Term	SL	RL (REINFORCE)
true label	$Y$	expected future rewards: $R\left(\tau \right)$
log term	$\log \left(\hat{Y}\right)$	log of policy: $\log {\pi }_{\theta }\left(a_{i,t}\mid s_{i,t}\right)$
cross entropy	$\frac{\sum _{i=1}^N{Y}_i\log {\hat{Y}}_i}{N}$	Monte Carlo expectation: $\frac{\sum _{i=1}^N\sum _{t=1}^T{R}_i\left(\tau \right)\log {\pi }_{\theta }\left(a_{i,t}\mid s_{i,t}\right)}{N}$

Table 2. Table 2: Comparing SL and RL for AAC methods

Term	SL	RL (A2C)
true label	$Y$	expected advantage: $A(s,a)=Q(s,a)-V\left(s\right)$
log term	$\log \left(\hat{Y}\right)$	log of policy: $\log {\pi }_{\theta }\left(a_{i,t}\mid s_{i,t}\right)$
cross entropy	$\frac{\sum _{i=1}^N{Y}_i\log {\hat{Y}}_i}{N}$	Monte Carlo expectation: $\frac{\sum _{i=1}^N\sum _{t=1}^{T}A(s_{i,t},a_{i,t})\log {\pi }_{\theta }\left(a_{i,t}\mid s_{i,t}\right)}{N}$

Equations

$$\tilde{J}(\theta )=\underset{N\to \infty }{\lim }\frac{1}{N}\sum _{i=1}^N\sum _{t=1}^T{R}_i(\tau )\log {\pi }_{\theta }(a_{i,t}\mid s_{i,t})$$

$$\tilde{J}(\theta )=\frac{\sum _{i=1}^N\sum _{t=1}^{T}A(s_{i,t},a_{i,t})\log {\pi }_{\theta }(a_{i,t}\mid s_{i,t})}{N}$$

$$H(p,q)={\mathbb{E}}_p[-\log q]$$

$$H(p,q)=H(p)+D_{\mathrm{KL}}(p\Vert q)$$

$$\mathbb{P}(\tau |\theta )=\mathbb{P}(s_1)\prod _{t=1}^T{\pi }_{\theta }(a_t\mid s_t)\mathbb{P}(s_{t+1}\mid s_t,a_t)$$

$$\log \mathbb{P}(\tau |\theta )$$

$${\nabla }_{\theta }\log \mathbb{P}(\tau |\theta )$$

$$J(\theta )={\mathbb{E}}_{\tau \sim \mathbb{P}(\tau |\theta )}\left[\sum _{t=1}^{T}r(s_t,a_t)\right]={\int }_{\tau }R(\tau )\mathbb{P}(\tau |\theta )d\tau$$

$${\nabla }_{\theta }J(\theta )$$

$${\nabla }_{\theta }J(\theta )=\mathbb{E}\left[R(\tau )\sum _{t=1}^T{\nabla }_{\theta }\log {\pi }_{\theta }(a_t\mid s_t)\right]$$

$${\nabla }_{\theta }J(\theta )=\underset{N\to \infty }{\lim }\frac{1}{N}\sum _{i=1}^N\sum _{t=1}^T{R}_i(\tau ){\nabla }_{\theta }\log {\pi }_{\theta }(a_{i,t}\mid s_{i,t})$$

$$\tilde{J}(\theta )=\underset{N\to \infty }{\lim }\frac{1}{N}\sum _{i=1}^N\sum _{t=1}^T{R}_i(\tau )\log {\pi }_{\theta }(a_{i,t}\mid s_{i,t})$$

Full text

keywords: Policy gradient, Supervised learning,

Cross entropy, Kullback Leibler divergence, entropy.

1 Introduction

In RL, PGM are frequently used Williams, (1992); Sutton et al., (1999); Silver et al., (2014); Lillicrap et al., (2015). They are traditionally opposed to value learning methods Sutton and Barto, (1998); Watkins and Dayan, (1992). PGM principle is very simple. Improve gradually the policy through gradient descent. PGM have two important concepts. It is a policy method as its name emphasizes. It is also a gradient descent method. Policy means we observe and act. Gradient descent methods uses the fact that the best move locally is along the gradient. As this moves is at first order, a learning rate needs to ensure policy improvement is not too large at each step. PGM have been popularized in REINFORCE Williams, (1992) and in Sutton et al., (1999) and have received wider attention with Actor Critic methods Konda and Tsitsiklis, (2003); Peters and Schaal, (2008) in particular when using deep PGM Mnih et al., (2016) that combines policy and value methods. In addition, recently, it has been found that an entropy regularization term may fasten convergence O’Donoghue et al., (2016); Nachum et al., (2017); Schulman et al., (2017).

When looking in details in REINFORCE Williams, (1992), we can remark that the gradient term with respect to the policy can indeed be interpreted as the log term in the cross entropy in supervised learning. If in addition, we make the bridge between RL and SL, emphasizing that RL problem can be reformulated as a SL problem where true labels are changed by expected discounted future rewards, and estimated probabilities by policy probabilities, the link between RL and SL becomes obvious. In addition, leveraging the tight relationship between cross entropy and Kullback Leibler divergence, we can interpret the entropy regularization terms very naturally. This is precisely the objective of this short paper. Call attention to the tight connection between RL and SL to give theoretical justification of some of the techniques used in PGMs.

The paper is organized as follows. In section 3, we recall the various choices of functional losses in SL and exhibit that cross entropy is one of the main possibilities for loss functions. We flaunt the relationship between cross entropy and Kullback Leibler divergence, harping on the additional entropy term. In section 4, we present PGMs. We rub in the interpretation of RL problems as a modified cross entropy SL problem. We conclude in section 5 with a financial numerical experience using deep PGM stressing that in the specific case of action that do not influence the environment, the difference between RL and SL may be very tenuous.

2 Related Work

Looking at similarities and synergies between RL and SL together was for a long time overlooked. This can be easily explained as the two research communities were different and thought they were labouring on incompatible or at least very different approaches. However, there has been one type of learning that promotes the similitude between RL and SL. This has been Imitation Learning.

Imitation learning (Schaal,, 1996) is a classic technique for learning from human demonstration. Imitation learning uses a supervised approach to imitate an expert’s behaviors, hence doing a RL task to accomplish a SL one. DAGGER (Ross et al.,, 2011) is considered to be the mainstream imitation algorithm. It requests an action from the expert at each step. It uses an action sampled from a mixed distribution from the agent and the expert. It combines the observed states and demonstrated actions to train the agent successively. This has led to numerous extensions of this algorithm and in particular to deep version of it. Deeply AggreVaTeD (Sun et al.,, 2017) extends DAGGER with deep neural networks and continuous action spaces. Other approaches have been to stop opposing RL and SL and rather leverage SL in deep RL. Hester et al., (2017) for instance have used SL tasks to train the agent better and created the method of Deep Q-learning from Demonstrations (DQfD). It tackles the problem of the necessity of huge amount of data for deep reinforcement learning (Deep RL) and the poor performance of Deep RL algorithms during initial phase of learning. The method of Deep Q-learning from Demonstrations (DQfD) solves the issue by combining RL techniques (temporal difference updates) with SL techniques (supervised classification of the demonstrator’s actions) as the target network in Deep Q learning is initially trained with supervised learning.

All of these works show that RL and SL are not as opposed as one may have thought. We argue here that as nice as these works are, they do not emphasizes that, ignoring for a while the issue of feedback effect of action on next state environment, a PGM can be reformulated as a SL task where true labels are changed into future expected reward while the PG can be interpreted as a cross entropy loss minimization.

3 Supervised Learning

The goal of SL classification is to infer a function from labelled training data that maps inputs into labelled outputs. In psychology, this is sometimes analyzed as concept learning. The deduction of the function parameters is done traditionally through the optimization of a loss function. SL classifiers parameters are the ones of the optimal solution of the optimization program. To keep things simple, let us assume that we are looking at a binary classification problem. Let us assume we observe $D_{n}=\{(X_{1},Y_{1}),\ldots,(X_{n},Y_{n})\}$ that are $n$ independent random copies of $(X,Y)\in\mathcal{X}\times\mathcal{Y}$. The feature $X$ lives in some abstract space $\mathcal{X}$ ($\mathbf{R}^{d}$ for instance) and $Y$ is called label. Binary classification assumes that $Y$ take two different values: $\mathcal{Y}=\{-1,1\}$, while multi-class classification assumes $\mathcal{Y}=\{1,\ldots,K\}$. To keep things simple, we will only look at binary classification. Naturally, one would like to find a function $f:\mathcal{X}\mapsto\mathbf{R}$ that best maps $X$ to $Y$. We are also given a loss function $\mathscr{l}:\{-1,1\}\times\{-1,1\}\mapsto\mathbf{R}$ that measures the error of a specific prediction. The loss function value at an arbitrary point $(Y,\hat{Y})$ reads as the cost incurred when predicting $\hat{Y}$ while true label is $Y$. In classification the loss function is often a zero-one loss, that is, $\mathscr{l}(Y,\hat{Y})$ is zero when the predicted label matches the true label $Y=\hat{Y}$ and one otherwise. To find our best classifier, we look for the classifier with the smallest expected loss. In other words, we look up for the function $f$ that minimizes the expected $\mathscr{l}$-risk, given by $\mathcal{R}_{\mathscr{l}}(h)=\mathbf{E}_{X\times Y}[\mathscr{l}(Y,f(X))]$.

Another naive approach is to minimize the empirical classification error $\mathbb{E}[\mathbbm{1}_{\{-Yf(X)\geq 0\}}]$. To bypass the non convexity of $\mathbbm{1}_{\mathbb{R}_{+}}$, we use convex risk minimization (CRM) Boucheron et al., (2005). CRM defines a convex surrogate for the classification problem, called the cost function $\varphi:\mathbb{R}\mapsto\mathbb{R}_{+}$ convex, non-decreasing such that $\varphi(0)=1$, hence $\varphi\geq\mathbbm{1}_{\mathbb{R}_{+}}$. The classification problem consists in minimizing the expected $\varphi$-risk : $\mathbb{E}[\varphi(-Yf(X))]$. Typical loss functions are

• •

square loss: $\varphi(u)=(1+u)^{2}$ for $u\geq 0$ and $\varphi(u)=1$ for $u\leq 0$, leading to regression methods.

• •

perplexity loss: $\varphi(u)=\log(e(1+u)))$ for $u>-1$ and $\varphi(u)=-\infty$ for $u\leq-1$.

• •

logit loss: $\varphi(u)=\log_{2}(1+e^{u})$, leading to logistic regression methods and cross entropy.

• •

hinge loss: $\varphi(u)=max(0,1+u)$. This leads to SVM methods.

• •

exponential loss: $\varphi(u)=e^{u}$.

We have also loss function defined directly between $Y$ and $\hat{Y}$ as follows:

• •

mean square error: $\mathscr{l}(Y,f(X))=(Y-f(X))^{2}$ that leads to standard regression methods.

• •

mean absolute error: $\mathscr{l}(Y,f(X))=|Y-f(X)|$.

• •

cross entropy: $\mathscr{l}(Y,f(X))=-\frac{1+Y}{2}\log\frac{1+f(X)}{2}$ leading to logistic regression with the usual convention $0\,\log 0\!=\!0$ justified by $\lim\limits_{x\to 0^{+}}x\log x=0$ (trivially obtained by L’Hopital’s rule).

• •

Huber loss: $\mathscr{l}(Y,f(X))=\frac{1}{2}(Y-f(X))^{2}$ if $|Y-f(X)|\leq\delta$ and $\mathscr{l}(Y,f(X))=\delta(|Y-f(X)|-\frac{1}{2}\delta^{2}$

We see from above that for SL there are numerous loss functions and that cross entropy is one criterium among others. We will see that this cross entropy has a nice interpretation in RL.

4 Reinforcement Learning Background

RL is usually modeled by an agent that interacts with an environment $\mathcal{E}$ over a number of discrete time steps. At each time step $t$, the agent levies a state $s_{t}$ and picks an action $a_{t}$ from a set of possible actions $\mathcal{A}$. This choice is done according to its policy $\pi$, where $\pi$ is a mapping from states $s_{t}$ to actions $a_{t}$. Once the action is decided and executed, the agent levies the next state $s_{t+1}$ and a scalar reward $r_{t}$. The goes on until the agent reaches a terminal state. The expected cumulated discounted return $R_{t}=\sum_{k=0}^{\infty}\gamma^{k}r_{t+k}$ is the sum of accumulated returns, where at each time step, future returns are discounted with the discount factor $\gamma\in(0,1]$. At time $t$, a rational agent seeks to maximize its expected return given his current state $s_{t}$.

We traditionally define

• •

the value of state $s$ under policy $\pi$ is defined as $V^{\pi}(s)=\mathbb{E}\left[R_{t}|s_{t}=s\right]$ and is simply the expected return for following policy $\pi$ from state $s$ ((Watkins,, 1989)).

• •

the action value function under policy $\pi$ $Q^{\pi}(s,a)=$ $\mathbb{E}\left[R_{t}|s_{t}=s,a\right]$ is defined as the expected return for selecting action $a$ in state $s$ and following policy $\pi$ ( Williams, (1992)).

Both the optimal value function $Q^{*}(s,a)=$ $\max_{\pi}Q^{\pi}(s,a)$ and the optimal value of state $V^{*}(s)=\max_{\pi}V^{\pi}(s)$ satisfy Bellmann equations.

Whenever states and action are too large, we are forced to represent the action value function with a function approximator, such as a neural network. Denoting the parameters $\theta$, the state action function writes $Q(s,a;\theta)$

The updates to $\theta$ can be derived from a variety of reinforcement learning algorithms. In particular, in value-based methods, policy-based model-free methods directly parameterize the policy $\pi(a|s;\theta)$ and update the parameters $\theta$ by performing, typically approximate, gradient ascent on $\mathbb{E}[R_{t}]$.

An illustration of such a method is REINFORCE due to  Williams, (1992). Standard REINFORCE updates the policy parameters $\theta$ in the direction R_{t}\nabla_{\theta}$$\log\pi(a_{t}|s_{t};\theta), which is an unbiased estimate of $\nabla_{\theta}\mathbb{E}[R_{t}]$. It is possible to reduce the variance of this estimate while keeping it unbiased by subtracting a learned function of the state $b_{t}(s_{t})$, known as a baseline (Williams,, 1992), from the return. The resulting gradient is $\nabla_{\theta}\log\pi(a_{t}|s_{t};\theta)\left(R_{t}-b_{t}(s_{t})\right)$. This approach can be viewed as an actor-critic architecture where the policy $\pi$ is the actor and the baseline $b_{t}$ is the critic(Sutton and Barto,, 1998; Degris et al.,, 2012).

We now prove a new formulation of REINFORCE.

Proposition 4.1**.**

The gradient descent in REINFORCE can also be computed by minimizing the following quantity

[TABLE]

For Advantage Actor Critic method, the gradient descent can also be computed by minimizing the following quantity:

[TABLE]

Proof.

See supplementary materials 7.1 ∎

It is enlightening to see that the two formulations, traditional reinforce and actor critic are very close to SL method with cross entropy. Recall that cross entropy is given by $Y\log(\hat{Y})$ for labels with value in $\{0,1\}$. This leads to the tables 1 and 2.

Last but not least, recall that there is a connection between cross entropy and Kullback Leibler divergence. Recall that the cross entropy for the distributions $p$ and $q$ over a given set is defined as follows:

[TABLE]

There is a straightforward connection to the Kullback–Leibler divergence $D_{\mathrm{KL}}(p\|q)$ of $q$ from $p$, sometimes referred to as the relative entropy of $p$ with respect to $q$ given by

[TABLE]

where $H(p)$ is the entropy of $p$. As the entropy term $H(p)$ is constant given the true distribution $p$, minimizing the cross entropy or the Kullback Leibler divergence is equivalent. However, for RL, this gives another nice interpretation. As shown previously, PGM can be cast as a cross entropy minimization program. Since the difference between Kullback Leibler divergence and cross entropy is this entropy term, it makes sense to incorporate this entropy term in our gradient descent optimization. In theory, the entropy term should be multiply by one . However in practice as the entropy term is estimated by the empirical entropy, it makes sense somehow to multiply the entropy term by a regularization term $\lambda$, leading to a variation of PGMs where instead of computing a gradient on cross entropy as in REINFORCE, we add an additional term called an entropy regularization. Somehow, this changes the minimization problem to something that is a modified Kullback Leibler divergence. The cross entropy term itself is computed on future expected reward. The analogy holds between SL and RL as long as actions do not influence or slightly influence the environment.

5 Experiments

We will apply our remark to a very specific environment where actions do not influence the environment. The considered reinforcement problem is a financial trading game concerning the Facebook stock (data were retrieved from https://finance.yahoo.com/quote/FB. We denote by $(P_{t})_{t=1,\ldots}$ the daily closing price of the Facebook stock in sequential order. For each day, we compute the daily return as follows $r_{t}=\frac{P_{t}}{P_{t-1}}-1$

The environment is composed of the $n$ last daily returns. We intentionally assumes $n$ last returns to emphasize that the choice of taking $n=5$ (last week) or $n=10$ (last two weeks) or $n=20$ (last month) is a model design decision. As returns are continuous, our state space is $\mathbb{R}_{+}^{n}$, which by RL standard is very large. Our possible actions are each day threefold: either do nothing, buy or sell the Facebook stock. If we decide to enter in a new position at time $t$, this will only be materialized the next day and hence we will be initially having an open position only at time $t+1$ initialized at the entering price $p_{t+1}$. Hence, if we only keep the position for one period, we will be facing the return $r_{t+2}$ as our position will be only closed at time $t+2$.

As for the reward, we take the Sharpe ratio of the trading strategy. As we compute the Sharpe ratio with daily returns, to compute the annual Sharpe ratio, we multiply the daily Sharpe ratio by a scaling factor equal to $\sqrt{250}$, assuming 250 trading days per year. We compute the daily Sharpe ratio as the mean of daily returns over their standard deviations. This implies in particular that we implicitly take a benchmark rate in the Sharpe ratio equal to 0.

To compute our Mark to Market (the value of our trading strategy), we mark any open position to the last know price. We parametrize our policy with a deep network consisting of two fully connected layers composed of 16 ReLU nodes. We keep the layer size as an hyper parameter and use initially REINFORCE and then A2C method to solve this reinforcement learning problem.

We provide the overall achieved Sharpe ratio with the Sharpe ratio reward choice for various learning rate and annealing rate. We obtained an overall Sharpe ratio higher than 1 which is quite a nice achievement compared to traditional investment strategies that typically do not perform better than a Sharpe ratio of 1.

We can notice in figure 1 that the overall Sharpe ratio is above 3 which is very high by financial market standards. This could be explained by the fact that the Facebook stock has been incredibly raising over the last five years. Hence the algorithm has not much difficulty finding the optimal strategy that is to buy and hold the stock

6 Conclusion

We show in this article that there are tight connections between SL and RL. PGM in RL can be cast as cross entropy minimization problems where true labels are replaced by expected future reward or advantage while the log term is changed into the log policy term. This analogy takes its root from the minimization problem where we are looking for the parameters that maximizes the expected futures reward or advantage. Should this optimization objective changed, we conjecture that we could make other analogies between SL and RL

7 Supplementary materials

7.1 Proof of proposition 4.1

We provide here a quick proof of REINFORCE with modern notations. Let us denote by $r(s_{t},a_{t})$ the reward for a state $s_{t}$ and action $a_{t}$. Let us assume we have some time horizon (that may be either finite or infinite). Let us denote by $\tau=(s_{1},a_{1},\ldots,s_{T},a_{T})$ a trajectory generated by our policy approximator governed by a parameter $\theta$. Using the Markov property of our MDP process, the probability of a given trajectory $\mathbb{P}(\tau|\theta)$ can be decomposed into a product of conditional probabilities as follows:

[TABLE]

Taking the log of the above equation (5), using the fact that the log of a product is the sum of the logs, we get:

[TABLE]

Differentiating with respect to theta gives (since most of terms do not depend on $\theta$):

[TABLE]

Recall we want to minimize the expected return, $J(\theta)$, defined as

[TABLE]

The above notation $\tau\sim\mathbb{P}(\tau|\theta)$ indicates that we’re sampling trajectories $\tau$ from the probability distribution of our policy approximator governed by $\theta$ and $R(\tau)$ is the sum of all the future (discounted) rewards.

To find the optimal $\theta$, we do gradient descent and hence need to compute

[TABLE]

Using the fact that we can interchange integral and expectation (9), assuming smooth functions and Lebesgue dominated convergence to justify that we can bring the gradient under the integral, we get:

[TABLE]

As the log gradient of a function is the quotient of the gradient and the function, we have:

[TABLE]

Expressed the integral as an expectation, we conclude:

[TABLE]

Finally, using the fact that the gradient of the log probability of the trajectory is the sum of the gradient of the log policy probabilities (7), we obtain the final expression:

[TABLE]

To turn this into something tractable, just express this as a Monte Carlo sum as follows:

[TABLE]

As the RHS does only depend on $\theta$ in the term $\log{\pi_{\theta}}(a_{i,t}\mid s_{i,t})$, minimizing $J(\theta)$ is the same as minimizing $\widetilde{J}(\theta)$ given by:

[TABLE]

As for Advantage Actor Critic, the above proof is the same and leads to the result. ∎