Robot Trains Robot: Automatic Real-World Policy Adaptation and Learning for Humanoids

TL;DR

This paper introduces RTR, a framework enabling efficient real-world reinforcement learning for humanoid robots through active guidance by a robotic arm teacher, addressing sim-to-real transfer challenges and reducing human intervention.

Contribution

The paper presents a novel RTR framework with a real-time RL pipeline and robotic teacher support, facilitating stable and efficient real-world humanoid policy learning.

Findings

Successful fine-tuning of humanoid walking policy for speed tracking

Learning a humanoid swing-up task from scratch in real-world

Demonstrated improved safety and efficiency in real-world training

Abstract

Simulation-based reinforcement learning (RL) has significantly advanced humanoid locomotion tasks, yet direct real-world RL from scratch or adapting from pretrained policies remains rare, limiting the full potential of humanoid robots. Real-world learning, despite being crucial for overcoming the sim-to-real gap, faces substantial challenges related to safety, reward design, and learning efficiency. To address these limitations, we propose Robot-Trains-Robot (RTR), a novel framework where a robotic arm teacher actively supports and guides a humanoid robot student. The RTR system provides protection, learning schedule, reward, perturbation, failure detection, and automatic resets. It enables efficient long-term real-world humanoid training with minimal human intervention. Furthermore, we propose a novel RL pipeline that facilitates and stabilizes sim-to-real transfer by optimizing a single dynamics-encoded latent variable in the real world. We validate our method through two challenging real-world humanoid tasks: fine-tuning a walking policy for precise speed tracking and learning a humanoid swing-up task from scratch, illustrating the promising capabilities of real-world humanoid learning realized by RTR-style systems. See https://robot-trains-robot.github.io/ for more info.