Learning to Walk via Deep Reinforcement Learning

TL;DR

This paper introduces a sample-efficient deep reinforcement learning algorithm based on maximum entropy RL, enabling a quadruped robot to learn stable walking gait directly in the real world within two hours without prior modeling or simulation.

Contribution

The authors develop a minimal-tuning, sample-efficient deep RL method that successfully learns real-world robotic locomotion from scratch, outperforming previous approaches in efficiency and robustness.

Findings

Learns stable walking gait in about two hours on real robot

Achieves state-of-the-art performance on simulated benchmarks

Policy is robust to environmental variations

Abstract

Deep reinforcement learning (deep RL) holds the promise of automating the acquisition of complex controllers that can map sensory inputs directly to low-level actions. In the domain of robotic locomotion, deep RL could enable learning locomotion skills with minimal engineering and without an explicit model of the robot dynamics. Unfortunately, applying deep RL to real-world robotic tasks is exceptionally difficult, primarily due to poor sample complexity and sensitivity to hyperparameters. While hyperparameters can be easily tuned in simulated domains, tuning may be prohibitively expensive on physical systems, such as legged robots, that can be damaged through extensive trial-and-error learning. In this paper, we propose a sample-efficient deep RL algorithm based on maximum entropy RL that requires minimal per-task tuning and only a modest number of trials to learn neural network policies. We apply this method to learning walking gaits on a real-world Minitaur robot. Our method can acquire a stable gait from scratch directly in the real world in about two hours, without relying on any model or simulation, and the resulting policy is robust to moderate variations in the environment. We further show that our algorithm achieves state-of-the-art performance on simulated benchmarks with a single set of hyperparameters. Videos of training and the learned policy can be found on the project website.