Learning walk and trot from the same objective using different types of exploration

TL;DR

This paper introduces a novel method for quadruped gait learning that encodes gait symmetry properties into the exploration process, enabling efficient learning of walk and trot gaits with improved performance.

Contribution

It proposes a new approach to incorporate gait symmetry into policy search, enhancing exploration and learning efficiency for quadruped locomotion.

Findings

Learned walk and trot gaits outperform random gaits.

Encoding symmetry improves exploration efficiency.

Performance significantly enhanced compared to baseline.

Abstract

In quadruped gait learning, policy search methods that scale high dimensional continuous action spaces are commonly used. In most approaches, it is necessary to introduce prior knowledge on the gaits to limit the highly non-convex search space of the policies. In this work, we propose a new approach to encode the symmetry properties of the desired gaits, on the initial covariance of the Gaussian search distribution, allowing for strategic exploration. Using episode-based likelihood ratio policy gradient and relative entropy policy search, we learned the gaits walk and trot on a simulated quadruped. Comparing these gaits to random gaits learned by initialized diagonal covariance matrix, we show that the performance can be significantly enhanced.

Equations

$$b_i(z)=\exp \left(\frac{\cos (2\pi (z_t-c_i))}{h}\right),$$

$$\mathbf{u}=K_p({\mathbf{q}}_{\mathrm{des}}-\mathbf{q})+K_D({\dot{\mathbf{q}}}_{\mathrm{des}}-\dot{\mathbf{q}}),$$

$${\mathbf{\mu }}^{\prime }=\mathbf{\mu }+\alpha {\nabla }_{\mathbf{\mu }}{J}_{\mathbf{\mu },\mathbf{\Sigma }}$$

$${\nabla }_{\mathbf{\mu }}{J}_{\mathbf{\mu },\mathbf{\Sigma }}$$

$$\approx \frac{1}{N}\sum _i^N{\mathbf{\Sigma }}^{-1}({\mathbf{\theta }}_i-\mathbf{\mu })(R({\mathbf{\theta }}_i)-\bar{R})$$

$$\begin{array}{rl}\underset{{\mathbf{\mu }}^{\prime },{\mathbf{\Sigma }}^{\prime }}{\max }{J}_{{\mathbf{\mu }}^{\prime },{\mathbf{\Sigma }}^{\prime }}=\max & {\int }_{\mathbf{\Theta }}p(\mathbf{\theta }|{\mathbf{\mu }}^{\prime },{\mathbf{\Sigma }}^{\prime })R(\mathbf{\theta })d\mathbf{\theta },\\ \text{s.t.}ϵ& \ge {\int }_{\mathbf{\Theta }}p(\mathbf{\theta }|{\mathbf{\mu }}^{\prime },{\mathbf{\Sigma }}^{\prime })\log \frac{p(\mathbf{\theta }|{\mathbf{\mu }}^{\prime },{\mathbf{\Sigma }}^{\prime })}{q(\mathbf{\theta }|\mathbf{\mu },\mathbf{\Sigma })},\\ 1& ={\int }_{\mathbf{\Theta }}p(\mathbf{\theta }|{\mathbf{\mu }}^{\prime },{\mathbf{\Sigma }}^{\prime })d\mathbf{\theta }.\end{array}$$

$$p(\mathbf{\theta }|{\mathbf{\mu }}^{\prime },{\mathbf{\Sigma }}^{\prime })\propto q(\mathbf{\theta }|\mathbf{\mu },\mathbf{\Sigma })\exp \left(\frac{R(\mathbf{\theta })}{\eta }\right),$$

$$g(\eta )\approx ϵ\eta +\eta \log \frac{1}{N}\sum _{i}q({\mathbf{\theta }}_{\mathbf{i}}|\mu ,\Sigma )\exp \left(\frac{R({\mathbf{\theta }}_{\mathbf{i}})}{\eta }\right)$$

$${\mathbf{\mu }}^{\prime }=\frac{{\sum }_{i=1}^N{d}_i{\mathbf{\mu }}_{\mathbf{i}}}{{\sum }_{i=1}^N{d}_i},{\mathbf{\Sigma }}^{\prime }=\frac{{\sum }_{i=1}^N{d}_i({\mathbf{\mu }}_{\mathbf{i}}-\mathbf{\mu })({\mu }_i-{\mathbf{\mu }}^T)}{Z},$$

$$Z=\frac{({\sum }_{i=1}^N{d}_i{)}^2-{\sum }_{i=1}^N(d_i{)}^2}{{\sum }_{i=1}^N{d}_i}.$$

$$R_{\tau }=\frac{1}{T}(\mathrm{sign}(y)\sqrt[]{x^2+y^2}-c_{\mathrm{ctrl}}(\mathbf{\tau })-c_{\mathrm{ct}}(\mathbf{\tau }))-c_{\mathrm{fall}}(\tau ),$$

$$\mathbf{\Sigma }={\sigma }^2(\mathbf{I}+\gamma \mathbf{O}),$$

$${\nabla }_{\mathbf{\mu }}{J}_{\mathbf{\mu },\mathbf{\Sigma }}$$

$$\approx \frac{1}{N}\sum _i^N{\nabla }_{\mathbf{\mu }}\log P({\mathbf{\tau }}_i|\mathbf{\mu },\mathbf{\Sigma })r({\mathbf{\tau }}_i)$$

$$=\frac{1}{N}\sum _i^N{\nabla }_{\mathbf{\mu }}\log \mathcal{N}({\mathbf{\theta }}_i|\mathbf{\mu },\mathbf{\Sigma })r({\mathbf{\theta }}_i)$$

$$=\frac{1}{N}\sum _i^N{\nabla }_{\mathbf{\mu }}(C-\frac{1}{2}({\mathbf{\theta }}_i-\mathbf{\mu }{)}^T{\mathbf{\Sigma }}^{-1}({\mathbf{\theta }}_i-\mathbf{\mu }))r({\mathbf{\theta }}_i)$$

$$=\frac{1}{N}\sum _i^N{\mathbf{\Sigma }}^{-1}({\mathbf{\theta }}_i-\mathbf{\mu })r({\mathbf{\theta }}_i)$$

$${\nabla }_{\mathbf{\mu }}{J}_{\mathbf{\mu },\mathbf{\Sigma }}\approx \frac{1}{N}\sum _i^N{\mathbf{\Sigma }}^{-1}({\mathbf{\theta }}_i-\mathbf{\mu })(r({\mathbf{\theta }}_i)-\bar{r})$$

$${\mathbf{w}}_i={\mathbf{P}}_1^{[i]}{\mathbf{w}}_1$$

$$\mathbf{\Sigma }=\mathbf{I}+\mathbf{O}=\underset{N\to \infty }{\lim }\frac{1}{N}\sum _i^N{\mathbf{w}}_{\mathrm{all},i}^T{\mathbf{w}}_{\mathrm{all},i},$$

Full text

1 Introduction

In nature, animals have developed extensive gaits to adapt to the different terrestrial terrain and situations, such as a horse galloping for faster speed, or a lizard trotting for a stable locomotion. In recent years, quadrupedal gait learning has attracted some research interest in robotics. Quadruped gaits offer a wide range of different movement patterns. As the cyclic movements of all four legs are similar, the gaits can be categorized mainly by the timing and order of the footfall, which can be represented as phase gaps among the trajectories of each leg.

In the presented work we learn open loop control policies for various gaits, focusing on walk and trot. In walk the leg trajectories are separated by quarter-phase gaps, resulting in an equidistant footfall, whereas in trot diagonal pairs of legs move synchronously and are separated by half-phase gaps. Other gaits that can be learned using the described approach are bound and pace. We show how these symmetry properties can be encoded in the parameter space of the chosen policy representation, in order to enhance the initial exploration and reliably learn the chosen gaits.

Neither do we fully define the gait in the policy representation as in [7, 8, 12, 3], nor do we learn random gaits [6] which could lead to a highly non convex problem. We only initialize the exploration. This way we can learn different gaits as a set of skills sharing a common representation, thus allowing for choosing gaits suitable to the current terrain.

Traditional reinforcement learning methods, such as TD-Learning, typically estimate the expected long-term reward at each state $\boldsymbol{s}$ and each time step $t$, to evaluate the quality of executing an action $\boldsymbol{u}$ in state $\boldsymbol{s}$. This is referred as the value function $V^{\pi}_{t}(\boldsymbol{s})$. Given a certain state and all possible actions, the action-value function $(\boldsymbol{s}_{t},\boldsymbol{u}_{t})$ is computed, an optimal action $\boldsymbol{u}^{*}_{t}$is then selected to optimize the action-value function [13]. However, this approach would be problematic in high dimensional state action space, since the whole space would need to be exhaustively covered. In addition, gait learning of a quadruped would also require continuous states and actions, meaning we would have to resort to value function approximation.

On the other hand, Policy Search methods offer the possibility to scale a high dimensional continuous action space to a reduced search space of possible policies. This can be done by parameterizing the search distribution $p{(\boldsymbol{\theta})}$, and directly operate and update in the parameter space $\boldsymbol{\Theta}$, $\boldsymbol{\theta}\in\boldsymbol{\Theta}$ [4]. In this work, we learn a Gaussian search distribution $p(\boldsymbol{\theta})=\mathcal{N}(\boldsymbol{\theta}|\boldsymbol{\mu,\Sigma})$ that maximizes the expected reward, using Likelihood Ratio Policy Gradient [15] and Relative Entropy Policy Search [10]. The trajectories of the quadruped are computed by a linear combination of the von Mises basis functions $\boldsymbol{b}$ and their corresponding weights $\boldsymbol{w}$, i.e., $\tau=\boldsymbol{b}^{T}\boldsymbol{w}$. The generated trajectories are fed to a PD controller to deterministically compute the motor torques.

2 Related Works

This section presents first the policy representation to be used in the experiments in conjunction with the subsequently presented episode-based policy search algorithms.

2.1 Policy Representation

The explored policies are central pattern generators using a linear representation $\boldsymbol{q}_{des}=\boldsymbol{b}^{T}\boldsymbol{w}$, $\boldsymbol{b}$ being the von Mises basis functions,

[TABLE]

where $h$ is the width, $c_{i}$ are the equidistant centers of basis function, $z(t)=\delta_{z}t$ is the phase, which is linear in time $t$ with a temporal scaling factor $\delta_{z}$ describing the speed of executing the trajectory [7, 9]. The policy parameters $\boldsymbol{\theta}$ to be learned then consist of the weights $\boldsymbol{w}$, and the temporal scaling factor $\delta_{z}$.

In order to simplify the learning problem, we hand tuned the initial standing posture of the quadruped to prevent it from falling. The learning exploration of each rollout begins then with the standing initialization.

The trajectories $\boldsymbol{q}_{\mathrm{des}}=\boldsymbol{\tau({\boldsymbol{\theta}})}$ defined by the learned policy, are then fed to a PD controller to deterministically compute the motor torques $\boldsymbol{u}$,

[TABLE]

where the desired velocities $\boldsymbol{\dot{q}}_{\mathrm{des}}$ are approximated using forward differences of $\boldsymbol{q}_{des}$, and $\boldsymbol{q}$, $\boldsymbol{\dot{q}}$ are the observed state variables.

2.2 Episode-based Policy Search Algorithms

Episode-based policy search algorithms typically maximize the expected reward ${J_{\boldsymbol{\mu},\boldsymbol{\Sigma}}}=\mathbb{E}_{\boldsymbol{\theta}}[R(\boldsymbol{\theta})]$ over a Gaussian search distribution $\mathcal{N}(\boldsymbol{\theta}|\boldsymbol{\mu},\boldsymbol{\Sigma})$, throughout the episode, w.r.t the distribution means $\boldsymbol{\mu}$ [4]. Independent of the policy search algorithm, the parameter vector $\boldsymbol{\theta}^{*}$, used for execution after a terminated learning process, is chosen as the final update of $\boldsymbol{\mu}$.

2.2.1 Likelihood Ratio Policy Gradient

Policy Gradient methods [11] use gradient ascent to maximize the expected return ${J}_{{\boldsymbol{\mu}},\boldsymbol{\Sigma}}$ over the search distribution w.r.t. the parameters ${\boldsymbol{\mu}}$,

[TABLE]

In the following we estimate the gradient according to the Likelihood Ratio Policy Gradient method (LRPG) [15] with the batch mean reward $\bar{R}$ as baseline, sampling from a Gaussian search distribution $\mathcal{N}\boldsymbol{\theta}_{i}|\boldsymbol{\mu},\boldsymbol{\Sigma})$. A derivation can be found in 5.1.

[TABLE]

2.2.2 Relative Entropy Policy Search

Similar to the Expectation Maximization (EM) algorithm [5], Relative Entropy Policy Search (REPS) also updates the mean and covariance of the search distribution to maximize the expected reward $J_{\boldsymbol{\mu,\Sigma}}$. In addition, REPS introduces the Kullback-Leibler divergence, as to upper bound the parameter update of the search distribution, thus forcing the parameter update to stay close to the sampled data. Considering the old distribution $q(\boldsymbol{\theta|\mu,\Sigma})$, and the newly estimated distribution $p(\boldsymbol{\theta|\mu^{\prime},\Sigma^{\prime}})$, the KL divergence is bounded by $\epsilon$, where $\text{KL}(q(\boldsymbol{\theta|\mu,\Sigma})|p(\boldsymbol{\theta|\mu^{\prime},\Sigma^{\prime}}))\leq\epsilon$. This then resolves to a constraint optimization problem

[TABLE]

The optimization can be solved by using the method of Lagrangian multipliers, resulting in a closed form solution of the new distribution

[TABLE]

where $\eta$ is obtained by minimizing the dual function $g(\eta)$, approximated by samples,

[TABLE]

The new distribution is then estimated by fitting a parametric distribution $p(\boldsymbol{\theta}|\boldsymbol{\mu}^{\prime},\boldsymbol{\Sigma^{\prime}})$ to the samples $\boldsymbol{\theta_{i}}$ and the corresponding reward $R(\boldsymbol{\theta_{i}})$. The parametric distribution is estimated using weighted maximum likelihood, where the weightings $d_{i}=\exp\left(\frac{R(\boldsymbol{(}\theta_{i}))}{\eta}\right)$

Given the Gaussian distribution of our set up, $p(\boldsymbol{\theta|\mu,\Sigma})$, the weighted maximum likelihood solution for updating the distribution, namely $\boldsymbol{\mu^{\prime}}$ and $\boldsymbol{\Sigma^{\prime}}$, is given by [5]

[TABLE]

where

[TABLE]

3 Evaluations

In this section, we evaluate the performance of the Likelihood Ratio Policy Gradient and the Relative Entropy Policy Search algorithm given different initializations for covariance of the Gaussian search distribution, which cause specific desired gaits to be learned.

3.1 Simulated Experimental Setup

We used the OpenAI Gym toolkit [2] with the MuJoCo simulator [14], and adapted the predefined ant environment to a quadruped with eight parallel joints. The quadruped is a complete symmetric robot, with four hip joints and four knee joints, where each joint is driven by an actuator, rendering in total eight degrees of freedom. Four basis functions are used for each joint, allowing for quarter-phase gaps, while reducing the amount of weights to learn. In total, the parameters to be learned consist of 32 weights $\boldsymbol{w}$, and the temporal scaling factor $\delta_{z}$.

Throughout the simulation, a batch size of 50 is used, and each learning episode consists of 200 time frames, equivalent to 2 seconds given that the time step is 0.01s. The number of policy updates is fixed to 60. 10 trials are carried out for each configuration to evaluate the robustness of the algorithms.

3.2 Reward Function and Algorithm Comparison

We devise the reward function consisting of the forward translation, costs of the robot falling over $c_{\mathrm{fall}}(\boldsymbol{\tau})$, contact costs between the feet and the ground $c_{\mathrm{ct}}(\boldsymbol{\tau})$, and the control costs for the motors $c_{\mathrm{ctrl}}(\boldsymbol{\tau})$

[TABLE]

where $x$ and $y$ denotes the total distance traveled in x and y direction in one episode, and $T$ is the duration of one episode. Since each learning process is initialized with a standing position, only one walking direction $\operatorname{sign}({y})$ is rewarded, which is the forward direction.

To compare the learning behavior of REPS and LRPG, both are evaluated with hand tuned hyper parameters, and diagonal covariance initializations $\boldsymbol{\Sigma}=\sigma^{2}\boldsymbol{I}$. A learning rate of $\alpha_{LRPG}=0.001$ is used for LRPG, and a KL bound of $\epsilon_{KL}=0.11$ for REPS. The learning curves of the expected reward and corresponding standard deviation are shown in Figure 1, where REPS learns relatively fast in the initial phase before converging to a stable solution. This is due to the applied KL bound, forcing the update to stay close to the old distribution [10]. However, REPS also suffers from premature convergence, since the updates of the covariance $\boldsymbol{\Sigma}$ reduce further exploration [1]. LRPG shows poor convergence properties, yet the exploration sometimes outperforms policies fond with REPS.

3.3 Different Initializations

Aside from a diagonal initialization of the search distributions covariance matrix, a customized covariance matrix can be used to incorporate prior knowledge about the symmetry properties of a specific gait,

[TABLE]

where $\sigma^{2}$ is the variance along each dimension, $\boldsymbol{O}$ is a symmetric matrix with elements $p_{ij}\in\{0;1\}$, and $\gamma\in(0,1)$ is the degree of coupling. This way the exploration is limited in certain search directions of the parameter space $\boldsymbol{\Theta}$, promoting the learning of movements with desired symmetry properties.

In Figure 2a and 2b three different initializations are applied: a diagonal and two non-diagonal matrices specific to walk and trot, where, the degree of coupling is $\gamma=0.9$ Each learning curve is averaged over 10 trials to evaluate the robustness of the policy updates, and the colored regions indicate the standard deviation of the expected reward over the 10 trials. In Figure 2b we can see that using the non-diagonal initializations, the slope of the learning curves as well as the expected reward of the learned policy increase compared to the diagonal initialization, indicating that the KL bounded updates of REPS are able to exploit the prior knowledge, even though it changes the covariance matrix in each update. In addition it shows that for this robot the trot gait is inherently faster than the walk gait, which is the usual case for most quadrupeds [16] In Figure 2a however, LRPG tends to show decreased performance for non-diagonal initializations.

Using LRPG as well as REPS, movements matching to the desired gaits are reliably learned given a degree of coupling for $\gamma>0.7$. In Figure 3, the snapshots of the learned walk and trot gaits are shown. The gaits are learned using REPS, initialized with the corresponding non-diagonal covariance matrices, specific to each gait. It can be seen that the learned gaits perform according to expected symmetry properties. In Figure 3a, one cycle of the walk gait is demonstrated. The legs lift individually in the order front right, hind left, front left, and hind right. In Figure 3b, one cycle of the trot gait is displayed. The front left and hind right leg lift up synchronously, as well as the font right and hind left leg. In between a standing posture can be observed.

3.4 Concrete Initializations

In this section we discuss how to create a template covariance matrix $\boldsymbol{\Sigma}=\boldsymbol{I}+\boldsymbol{O}$ given the phase gaps specific to a desired gait. Note that this template only indicates the non-zero elements of the scaled covariance matrix defined in (11), excluding the parts corresponding to the temporal scaling $\delta_{z}$ which is always diagonal. We assume a maximal coupling of $\gamma=1$, reducing the exploration to a subspace of $\boldsymbol{\Theta}$. In this case the coupling of the movement is identical to the coupling of the exploration, as no parameters outside of the subspace are sampled.

In Figure 4 the initializations of the covariance matrix $\boldsymbol{\Sigma}$ is displayed as a heat map, divided into 8x8 blocks, corresponding to the coupling of the 8 weights describing one leg trajectory to another. Each of the blocks can further be divided into 4x4 sub-blocks corresponding to the joint-wise coupling. In our setup the knees are not coupled with the hips, therefore only the 4x4 sub-blocks on the diagonal of each 8x8 block are non-zero. Using 4 identically distributed cyclic basis functions per joint, any multiple of quarter-phase gaps, can be expressed as a permutation of weights. All 4x4 sub-blocks are identical to the permutation matrix that applies the desired multiple of quarter-phase gaps, which in our case are shifts of the weights. Since the phase gaps of the knee trajectories and hip trajectories are the same for each pair of legs, all 8x8 blocks consist of by two identical non-zero 4x4 sub-blocks. Thus, as only four different quarter-phase gaps can be defined, there can only be four different 8x8 blocks.

In Figure 4b, it can be observed that leg 1 and leg 2 are coupled to move synchronously with a half-phase gap to leg 3 and leg 4. Whereas in Figure 4a, all four different 8x8 blocks can be observed, as well as the individual leg movements being delayed by quarter-phase gaps.

3.5 Change of Exploration in REPS

Since each iteration of REPS considerably changes the covariance of the search distribution, it diverges from its initialization within a few policy updates. As seen in Figure 5a, up until the third update, the structure of the initialization for walk gait is preserved, however as shown in Figure 5b, the structure is lost at the fourth update. Nevertheless, the custom exploration during the first few policy evaluations would still strongly impact the following learning process.

4 CONCLUSION

In this paper we presented a new approach to introduce prior knowledge in the context of policy search reinforcement learning, using the policy representation of von Mises basis functions. Namely we initialized the covariance of a Gaussian search distribution, and applied it to the problem of quadruped gait learning. We showed that, using REPS, significant improvements in performance can be gained by learning specific types of gaits, i.e., walk and trot compared to random gaits initialized by a diagonal covariance matrix. The different types of gaits can be learned sharing the same linear basis function representation. Further more, REPS has shown to have a more stable learning process, albeit converging prematurely, whereas LRPG shows a slower and unstable learning process.

5 APPENDIX

5.1 Derivation of the Likelihood Ratio Policy Gradient

The expectation is approximated by the sampling. The individual trajectories $\boldsymbol{\tau}_{i}$ are sufficiently described by the policies parameters $\boldsymbol{\theta}_{i}$ for an open-loop control. To reduce variance, the mean reward of each batch $\bar{R}$ is used as baseline.

[TABLE]

[TABLE]

5.2 Estimation of covariance templates by sampling

The sub matrix corresponding to the weights in linear function representation is estimated by sampling. Three permutation matrices are devised: $\boldsymbol{P}^{[2]}_{1}$, $\boldsymbol{P}_{1}^{[3]}$, and $\boldsymbol{P}_{1}^{[4]}$, which define the coupling of the weight vectors $\boldsymbol{w}_{i}$ assigned to each leg w.r.t. leg 1,

[TABLE]

When sampling the weights of the first leg from a gaussian distribution $\boldsymbol{w}_{1,i}\sim\mathcal{N}(\boldsymbol{0},\boldsymbol{I})$ with zero mean and identity covariance the covariance template can be calculated as

[TABLE]

with $\boldsymbol{w}_{\mathrm{all},i}=(\boldsymbol{w}_{1,i},\boldsymbol{w}_{2,i},\boldsymbol{w}_{3,i},\boldsymbol{w}_{4,i})^{T}$. The limit can be calculated for sufficiently large $N$ by rounding.