Sim-to-Real Learning of Footstep-Constrained Bipedal Dynamic Walking

TL;DR

This paper presents a reinforcement learning approach for bipedal robots that maintains dynamic, robust gaits while respecting external footstep constraints, demonstrated through sim-to-real transfer on the Cassie robot.

Contribution

It introduces an RL formulation for footstep-constrained gait control and a supervised learning model for predicting feasible touchdown locations.

Findings

Successful sim-to-real transfer on Cassie robot

Effective response to externally imposed footstep constraints

Accurate prediction of next touchdown locations

Abstract

Recently, work on reinforcement learning (RL) for bipedal robots has successfully learned controllers for a variety of dynamic gaits with robust sim-to-real demonstrations. In order to maintain balance, the learned controllers have full freedom of where to place the feet, resulting in highly robust gaits. In the real world however, the environment will often impose constraints on the feasible footstep locations, typically identified by perception systems. Unfortunately, most demonstrated RL controllers on bipedal robots do not allow for specifying and responding to such constraints. This missing control interface greatly limits the real-world application of current RL controllers. In this paper, we aim to maintain the robust and dynamic nature of learned gaits while also respecting footstep constraints imposed externally. We develop an RL formulation for training dynamic gait controllers that can respond to specified touchdown locations. We then successfully demonstrate simulation and sim-to-real performance on the bipedal robot Cassie. In addition, we use supervised learning to induce a transition model for accurately predicting the next touchdown locations that the controller can achieve given the robot's proprioceptive observations. This model paves the way for integrating the learned controller into a full-order robot locomotion planner that robustly satisfies both balance and environmental constraints.