Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids

TL;DR

This paper presents a practical sim-to-real reinforcement learning approach enabling humanoid robots to perform complex vision-based dexterous manipulation tasks with high success and robustness, demonstrating scalability and real-world applicability.

Contribution

The paper introduces a novel sim-to-real RL framework with automated tuning, generalized reward, policy distillation, and hybrid object representation for humanoid manipulation.

Findings

High success rates on unseen objects

Robust and adaptive manipulation behaviors

Scalable to real-world humanoid tasks

Abstract

Learning generalizable robot manipulation policies, especially for complex multi-fingered humanoids, remains a significant challenge. Existing approaches primarily rely on extensive data collection and imitation learning, which are expensive, labor-intensive, and difficult to scale. Sim-to-real reinforcement learning (RL) offers a promising alternative, but has mostly succeeded in simpler state-based or single-hand setups. How to effectively extend this to vision-based, contact-rich bimanual manipulation tasks remains an open question. In this paper, we introduce a practical sim-to-real RL recipe that trains a humanoid robot to perform three challenging dexterous manipulation tasks: grasp-and-reach, box lift and bimanual handover. Our method features an automated real-to-sim tuning module, a generalized reward formulation based on contact and object goals, a divide-and-conquer policy distillation framework, and a hybrid object representation strategy with modality-specific augmentation. We demonstrate high success rates on unseen objects and robust, adaptive policy behaviors -- highlighting that vision-based dexterous manipulation via sim-to-real RL is not only viable, but also scalable and broadly applicable to real-world humanoid manipulation tasks.