Learning Multi-Modal Whole-Body Control for Real-World Humanoid Robots

TL;DR

This paper presents the Masked Humanoid Controller (MHC), a unified learned control interface for diverse whole-body behaviors in humanoid robots, trained in simulation and effective in real-world execution.

Contribution

Introduction of MHC, a flexible, multi-modal, learned control framework for humanoid robots that handles diverse behaviors via a unified interface.

Findings

MHC enables robust execution of diverse behaviors in simulation.

MHC successfully controls a real humanoid robot in various tasks.

The unified interface simplifies command specification across modalities.

Abstract

A major challenge in humanoid robotics is designing a unified interface for commanding diverse whole-body behaviors, from precise footstep sequences to partial-body mimicry and joystick teleoperation. We introduce the Masked Humanoid Controller (MHC), a learned whole-body controller that exposes a simple yet expressive interface: the specification of masked target trajectories over selected subsets of the robot's state variables. This unified abstraction allows high-level systems to issue commands in a flexible format that accommodates multi-modal inputs such as optimized trajectories, motion capture clips, re-targeted video, and real-time joystick signals. The MHC is trained in simulation using a curriculum that spans this full range of modalities, enabling robust execution of partially specified behaviors while maintaining balance and disturbance rejection. We demonstrate the MHC both in simulation and on the real-world Digit V3 humanoid, showing that a single learned controller is capable of executing such diverse whole-body commands in the real world through a common representational interface.