Learning Memory-Based Control for Human-Scale Bipedal Locomotion

TL;DR

This paper explores the use of recurrent neural networks for controlling bipedal robots, demonstrating improved simulation performance and better transfer to real robots through dynamics randomization and internal memory utilization.

Contribution

It introduces RNN-based policies for bipedal locomotion, highlighting the importance of dynamics randomization and memory for effective sim-to-real transfer.

Findings

RNNs outperform memoryless policies in simulation.

Dynamics randomization improves real-world transfer.

RNNs can encode system parameters for online identification.

Abstract

Controlling a non-statically stable biped is a difficult problem largely due to the complex hybrid dynamics involved. Recent work has demonstrated the effectiveness of reinforcement learning (RL) for simulation-based training of neural network controllers that successfully transfer to real bipeds. The existing work, however, has primarily used simple memoryless network architectures, even though more sophisticated architectures, such as those including memory, often yield superior performance in other RL domains. In this work, we consider recurrent neural networks (RNNs) for sim-to-real biped locomotion, allowing for policies that learn to use internal memory to model important physical properties. We show that while RNNs are able to significantly outperform memoryless policies in simulation, they do not exhibit superior behavior on the real biped due to overfitting to the simulation physics unless trained using dynamics randomization to prevent overfitting; this leads to consistently better sim-to-real transfer. We also show that RNNs could use their learned memory states to perform online system identification by encoding parameters of the dynamics into memory.