SPIDER: Scalable Physics-Informed Dexterous Retargeting

TL;DR

SPIDER is a physics-informed retargeting framework that transforms human demonstrations into dynamically feasible robot trajectories, significantly improving data efficiency and success rates for policy learning in dexterous robotics.

Contribution

It introduces a scalable, physics-based retargeting method that bridges the gap between human demonstrations and robot dynamics, enabling large-scale data augmentation for policy learning.

Findings

Improves success rates by 18% over standard sampling

10X faster than reinforcement learning baselines

Generates a 2.4 million frame dataset of feasible robot trajectories

Abstract

Learning dexterous and agile policy for humanoid and dexterous hand control requires large-scale demonstrations, but collecting robot-specific data is prohibitively expensive. In contrast, abundant human motion data is readily available from motion capture, videos, and virtual reality, which could help address the data scarcity problem. However, due to the embodiment gap and missing dynamic information like force and torque, these demonstrations cannot be directly executed on robots. To bridge this gap, we propose Scalable Physics-Informed DExterous Retargeting (SPIDER), a physics-based retargeting framework to transform and augment kinematic-only human demonstrations to dynamically feasible robot trajectories at scale. Our key insight is that human demonstrations should provide global task structure and objective, while large-scale physics-based sampling with curriculum-style virtual contact guidance should refine trajectories to ensure dynamical feasibility and correct contact sequences. SPIDER scales across diverse 9 humanoid/dexterous hand embodiments and 6 datasets, improving success rates by 18% compared to standard sampling, while being 10X faster than reinforcement learning (RL) baselines, and enabling the generation of a 2.4M frames dynamic-feasible robot dataset for policy learning. As a universal physics-based retargeting method, SPIDER can work with diverse quality data and generate diverse and high-quality data to enable efficient policy learning with methods like RL.