Learning Human-to-Humanoid Real-Time Whole-Body Teleoperation

TL;DR

This paper introduces H2O, a reinforcement learning framework that enables real-time, whole-body teleoperation of humanoid robots from RGB camera input, using a scalable simulation-to-data approach for motion learning.

Contribution

It presents a novel scalable sim-to-data process and a real-time RL-based method for human-to-humanoid teleoperation, achieving dynamic motions in real-world scenarios.

Findings

Successful real-time teleoperation of humanoid robots with diverse motions

Zero-shot transfer from simulation to real robot

First demonstration of learning-based whole-body humanoid teleoperation

Abstract

We present Human to Humanoid (H2O), a reinforcement learning (RL) based framework that enables real-time whole-body teleoperation of a full-sized humanoid robot with only an RGB camera. To create a large-scale retargeted motion dataset of human movements for humanoid robots, we propose a scalable "sim-to-data" process to filter and pick feasible motions using a privileged motion imitator. Afterwards, we train a robust real-time humanoid motion imitator in simulation using these refined motions and transfer it to the real humanoid robot in a zero-shot manner. We successfully achieve teleoperation of dynamic whole-body motions in real-world scenarios, including walking, back jumping, kicking, turning, waving, pushing, boxing, etc. To the best of our knowledge, this is the first demonstration to achieve learning-based real-time whole-body humanoid teleoperation.