Sim-to-Real Learning of All Common Bipedal Gaits via Periodic Reward Composition

TL;DR

This paper introduces a novel reward composition framework for sim-to-real reinforcement learning that enables a bipedal robot to learn and transition between all common gaits, such as walking and running, with reliable transfer from simulation to reality.

Contribution

It proposes a probabilistic periodic reward framework that simplifies gait specification and improves sim-to-real transfer for diverse bipedal locomotion behaviors.

Findings

Successful transfer of multiple gaits to real robot Cassie

A unified reward function for all common bipedal gaits

A policy capable of transitioning between different gaits

Abstract

We study the problem of realizing the full spectrum of bipedal locomotion on a real robot with sim-to-real reinforcement learning (RL). A key challenge of learning legged locomotion is describing different gaits, via reward functions, in a way that is intuitive for the designer and specific enough to reliably learn the gait across different initial random seeds or hyperparameters. A common approach is to use reference motions (e.g. trajectories of joint positions) to guide learning. However, finding high-quality reference motions can be difficult and the trajectories themselves narrowly constrain the space of learned motion. At the other extreme, reference-free reward functions are often underspecified (e.g. move forward) leading to massive variance in policy behavior, or are the product of significant reward-shaping via trial-and-error, making them exclusive to specific gaits. In this work, we propose a reward-specification framework based on composing simple probabilistic periodic costs on basic forces and velocities. We instantiate this framework to define a parametric reward function with intuitive settings for all common bipedal gaits - standing, walking, hopping, running, and skipping. Using this function we demonstrate successful sim-to-real transfer of the learned gaits to the bipedal robot Cassie, as well as a generic policy that can transition between all of the two-beat gaits.