Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning

TL;DR

This paper demonstrates that Deep Reinforcement Learning can be used to teach a miniature humanoid robot complex, agile soccer skills, enabling it to perform safe, adaptive, and strategic behaviors in dynamic environments with successful simulation-to-real transfer.

Contribution

The study introduces a novel application of Deep RL for training a humanoid robot to perform complex soccer skills, including strategic behaviors, with successful zero-shot transfer from simulation to real robots.

Findings

Robots achieved 181% faster walking speed

Turned 302% faster than baseline

Kicked a ball 34% faster than scripted methods

Abstract

We investigate whether Deep Reinforcement Learning (Deep RL) is able to synthesize sophisticated and safe movement skills for a low-cost, miniature humanoid robot that can be composed into complex behavioral strategies in dynamic environments. We used Deep RL to train a humanoid robot with 20 actuated joints to play a simplified one-versus-one (1v1) soccer game. The resulting agent exhibits robust and dynamic movement skills such as rapid fall recovery, walking, turning, kicking and more; and it transitions between them in a smooth, stable, and efficient manner. The agent's locomotion and tactical behavior adapts to specific game contexts in a way that would be impractical to manually design. The agent also developed a basic strategic understanding of the game, and learned, for instance, to anticipate ball movements and to block opponent shots. Our agent was trained in simulation and transferred to real robots zero-shot. We found that a combination of sufficiently high-frequency control, targeted dynamics randomization, and perturbations during training in simulation enabled good-quality transfer. Although the robots are inherently fragile, basic regularization of the behavior during training led the robots to learn safe and effective movements while still performing in a dynamic and agile way -- well beyond what is intuitively expected from the robot. Indeed, in experiments, they walked 181% faster, turned 302% faster, took 63% less time to get up, and kicked a ball 34% faster than a scripted baseline, while efficiently combining the skills to achieve the longer term objectives.