Learning Sim-to-Real Humanoid Locomotion in 15 Minutes

TL;DR

This paper presents a practical method to train humanoid locomotion policies in just 15 minutes using off-policy RL algorithms, massively parallel simulation, and minimal reward functions, enabling rapid sim-to-real transfer.

Contribution

The authors introduce a simple recipe with off-policy RL algorithms and careful design choices that achieves fast, stable training of humanoid control policies in minutes.

Findings

Training time reduced to 15 minutes on a single GPU.

Successful transfer of policies to real humanoid robots.

Effective handling of domain randomization and perturbations.

Abstract

Massively parallel simulation has reduced reinforcement learning (RL) training time for robots from days to minutes. However, achieving fast and reliable sim-to-real RL for humanoid control remains difficult due to the challenges introduced by factors such as high dimensionality and domain randomization. In this work, we introduce a simple and practical recipe based on off-policy RL algorithms, i.e., FastSAC and FastTD3, that enables rapid training of humanoid locomotion policies in just 15 minutes with a single RTX 4090 GPU. Our simple recipe stabilizes off-policy RL algorithms at massive scale with thousands of parallel environments through carefully tuned design choices and minimalist reward functions. We demonstrate rapid end-to-end learning of humanoid locomotion controllers on Unitree G1 and Booster T1 robots under strong domain randomization, e.g., randomized dynamics, rough terrain, and push perturbations, as well as fast training of whole-body human-motion tracking policies. We provide videos and open-source implementation at: https://younggyo.me/fastsac-humanoid.