Hierarchical Reinforcement Learning and Value Optimization for Challenging Quadruped Locomotion

TL;DR

This paper introduces a hierarchical reinforcement learning framework for quadruped robots that improves navigation over difficult terrains by combining high-level goal setting with low-level footstep control, using value optimization without extra training.

Contribution

A novel hierarchical RL approach that leverages online value optimization for quadruped locomotion, eliminating the need for additional training of the high-level policy.

Findings

Achieves higher rewards and fewer collisions than end-to-end RL methods.

Effective on terrains more challenging than those used during training.

Operates via online optimization without extra environment samples.

Abstract

We propose a novel hierarchical reinforcement learning framework for quadruped locomotion over challenging terrain. Our approach incorporates a two-layer hierarchy in which a high-level policy (HLP) selects optimal goals for a low-level policy (LLP). The LLP is trained using an on-policy actor-critic RL algorithm and is given footstep placements as goals. We propose an HLP that does not require any additional training or environment samples and instead operates via an online optimization process over the learned value function of the LLP. We demonstrate the benefits of this framework by comparing it with an end-to-end reinforcement learning (RL) approach. We observe improvements in its ability to achieve higher rewards with fewer collisions across an array of different terrains, including terrains more difficult than any encountered during training.