Robust Feedback Motion Policy Design Using Reinforcement Learning on a 3D Digit Bipedal Robot

TL;DR

This paper introduces a hierarchical reinforcement learning framework with feedback control for robust bipedal walking on the Digit robot, achieving successful sim-to-real transfer without extensive tuning or curriculum learning.

Contribution

The authors develop a cascade-structure controller integrating feedback regulation with reinforcement learning, enabling stable, robust walking gait transfer from simulation to hardware with minimal tuning.

Findings

Successful transfer of learned policy to Digit robot hardware

Robust walking gait under external disturbances and challenging terrains

Reduced sampling efficiency due to simplified state and action spaces

Abstract

In this paper, a hierarchical and robust framework for learning bipedal locomotion is presented and successfully implemented on the 3D biped robot Digit built by Agility Robotics. We propose a cascade-structure controller that combines the learning process with intuitive feedback regulations. This design allows the framework to realize robust and stable walking with a reduced-dimension state and action spaces of the policy, significantly simplifying the design and reducing the sampling efficiency of the learning method. The inclusion of feedback regulation into the framework improves the robustness of the learned walking gait and ensures the success of the sim-to-real transfer of the proposed controller with minimal tuning. We specifically present a learning pipeline that considers hardware-feasible initial poses of the robot within the learning process to ensure the initial state of the learning is replicated as close as possible to the initial state of the robot in hardware experiments. Finally, we demonstrate the feasibility of our method by successfully transferring the learned policy in simulation to the Digit robot hardware, realizing sustained walking gaits under external force disturbances and challenging terrains not included during the training process. To the best of our knowledge, this is the first time a learning-based policy is transferred successfully to the Digit robot in hardware experiments without using dynamic randomization or curriculum learning.