HumDex: Humanoid Dexterous Manipulation Made Easy

TL;DR

HumDex is a portable teleoperation system for humanoid robots that combines IMU-based motion tracking and learning-based hand retargeting, enabling efficient data collection and improved generalization in dexterous manipulation tasks.

Contribution

The paper introduces HumDex, a novel portable teleoperation system with a learning-based hand retargeting method and a two-stage imitation learning framework for humanoid manipulation.

Findings

Enhanced accuracy in full-body tracking using IMU sensors.

Improved generalization to new tasks and environments with minimal data.

Open-source implementation for reproducibility.

Abstract

This paper investigates humanoid whole-body dexterous manipulation, where the efficient collection of high-quality demonstration data remains a central bottleneck. Existing teleoperation systems often suffer from limited portability, occlusion, or insufficient precision, which hinders their applicability to complex whole-body tasks. To address these challenges, we introduce HumDex, a portable teleoperation system designed for humanoid whole-body dexterous manipulation. Our system leverages IMU-based motion tracking to address the portability-precision trade-off, enabling accurate full-body tracking while remaining easy to deploy. For dexterous hand control, we further introduce a learning-based retargeting method that generates smooth and natural hand motions without manual parameter tuning. Beyond teleoperation, HumDex enables efficient collection of human motion data. Building on this capability, we propose a two-stage imitation learning framework that first pre-trains on diverse human motion data to learn generalizable priors, and then fine-tunes on robot data to bridge the embodiment gap for precise execution. We demonstrate that this approach significantly improves generalization to new configurations, objects, and backgrounds with minimal data acquisition costs. The entire system is fully reproducible and open-sourced at https://github.com/physical-superintelligence-lab/humdex.