DexSynRefine: Synthesizing and Refining Human-Object Interaction Motion for Physically Feasible Dexterous Robot Actions

TL;DR

DexSynRefine is a comprehensive framework that synthesizes and refines human-object interaction motions to enable physically feasible and transferable dexterous robot actions, addressing embodiment mismatch and contact dynamics.

Contribution

It introduces a novel motion synthesis and refinement framework combining motion primitives, residual RL, and contact adaptation for improved robot manipulation.

Findings

Outperforms prior baselines in simulation across five tasks.

Successfully transfers to real robots with significant performance gains.

Enhances motion synthesis from sparse human demonstrations.

Abstract

Learning dexterous manipulation from human-object interaction (HOI) data is a scalable alternative to teleoperation, but HOI demonstrations are sparse and provide only kinematic motion that is not directly executable under embodiment mismatch and contact-rich dynamics. We present DexSynRefine, a framework with three coupled components: HOI-MMFP, a task- and object-initial-state-conditioned extension of motion manifold primitives that synthesizes coordinated hand-object trajectories from sparse HOI demonstrations; a task-space residual RL policy that physically grounds the synthesized reference while inheriting its kinematic structure; and a contact-and-dynamics adaptation module that enables sim-to-real transfer from proprioceptive history. Across five dexterous manipulation tasks spanning pick-and-place, tool use, and object reorientation, our task-space residual policy outperforms prior action-representation baselines in simulations and transfers to a real robot on all five tasks, improving over kinematic retargeting by 50-70 percentage points.