MOSAIC: Bridging the Sim-to-Real Gap in Generalist Humanoid Motion Tracking and Teleoperation with Rapid Residual Adaptation

TL;DR

MOSAIC is a comprehensive system that enhances humanoid motion tracking and teleoperation by bridging the sim-to-real gap through rapid residual adaptation, enabling robust real-world performance across various interfaces.

Contribution

It introduces a novel rapid residual adaptation method that improves general humanoid motion tracking and teleoperation across multiple interfaces.

Findings

Outperforms naive fine-tuning and continual learning methods.

Demonstrates robust offline motion replay and online long-horizon teleoperation.

Validates effectiveness through systematic ablations and real-robot experiments.

Abstract

Generalist humanoid motion trackers have recently achieved strong simulation metrics by scaling data and training, yet often remain brittle on hardware during sustained teleoperation due to interface- and dynamics-induced errors. We present MOSAIC, an open-source, full-stack system for humanoid motion tracking and whole-body teleoperation across multiple interfaces. MOSAIC first learns a teleoperation-oriented general motion tracker via RL on a multi-source motion bank with adaptive resampling and rewards that emphasize world-frame motion consistency, which is critical for mobile teleoperation. To bridge the sim-to-real interface gap without sacrificing generality, MOSAIC then performs rapid residual adaptation: an interface-specific policy is trained using minimal interface-specific data, and then distilled into the general tracker through an additive residual module, outperforming naive fine-tuning or continual learning. We validate MOSAIC with systematic ablations, out-of-distribution benchmarking, and real-robot experiments demonstrating robust offline motion replay and online long-horizon teleoperation under realistic latency and noise. Project page: baai-humanoid.github.io/MOSAIC.