Learning Diverse Bimanual Dexterous Manipulation Skills from Human Demonstrations

TL;DR

This paper introduces BiDexHD, a novel framework that learns diverse bimanual manipulation skills from human demonstrations, enabling scalable, multi-task, and zero-shot generalization in robotic dexterous manipulation.

Contribution

The work presents a unified task construction method and a teacher-student policy learning approach for efficient multi-task learning from demonstrations.

Findings

Achieved 74.59% task fulfillment on trained tasks.

Attained 51.07% success rate on unseen tasks.

Demonstrated effective zero-shot generalization capabilities.

Abstract

Bimanual dexterous manipulation is a critical yet underexplored area in robotics. Its high-dimensional action space and inherent task complexity present significant challenges for policy learning, and the limited task diversity in existing benchmarks hinders general-purpose skill development. Existing approaches largely depend on reinforcement learning, often constrained by intricately designed reward functions tailored to a narrow set of tasks. In this work, we present a novel approach for efficiently learning diverse bimanual dexterous skills from abundant human demonstrations. Specifically, we introduce BiDexHD, a framework that unifies task construction from existing bimanual datasets and employs teacher-student policy learning to address all tasks. The teacher learns state-based policies using a general two-stage reward function across tasks with shared behaviors, while the student distills the learned multi-task policies into a vision-based policy. With BiDexHD, scalable learning of numerous bimanual dexterous skills from auto-constructed tasks becomes feasible, offering promising advances toward universal bimanual dexterous manipulation. Our empirical evaluation on the TACO dataset, spanning 141 tasks across six categories, demonstrates a task fulfillment rate of 74.59% on trained tasks and 51.07% on unseen tasks, showcasing the effectiveness and competitive zero-shot generalization capabilities of BiDexHD. For videos and more information, visit our project page https://sites.google.com/view/bidexhd.