UniHM: Unified Dexterous Hand Manipulation with Vision Language Model

TL;DR

UniHM is a novel framework enabling dexterous hand manipulation guided by open-vocabulary language commands, using a unified tokenizer, language-conditioned models, and physics-based refinement for realistic, generalizable robotic manipulation.

Contribution

It introduces a unified hand tokenizer, a vision-language action model trained on interaction data, and a physics-guided refinement module, advancing open-vocabulary dexterous manipulation.

Findings

Achieves state-of-the-art results on multiple datasets.

Demonstrates strong generalization to unseen objects and trajectories.

Produces smooth, physically feasible manipulation sequences.

Abstract

Planning physically feasible dexterous hand manipulation is a central challenge in robotic manipulation and Embodied AI. Prior work typically relies on object-centric cues or precise hand-object interaction sequences, foregoing the rich, compositional guidance of open-vocabulary instruction. We introduce UniHM, the first framework for unified dexterous hand manipulation guided by free-form language commands. We propose a Unified Hand-Dexterous Tokenizer that maps heterogeneous dexterous-hand morphologies into a single shared codebook, improving cross-dexterous hand generalization and scalability to new morphologies. Our vision language action model is trained solely on human-object interaction data, eliminating the need for massive real-world teleoperation datasets, and demonstrates strong generalizability in producing human-like manipulation sequences from open-ended language instructions. To ensure physical realism, we introduce a physics-guided dynamic refinement module that performs segment-wise joint optimization under generative and temporal priors, yielding smooth and physically feasible manipulation sequences. Across multiple datasets and real-world evaluations, UniHM attains state-of-the-art results on both seen and unseen objects and trajectories, demonstrating strong generalization and high physical feasibility. Our project page at \href{https://unihm.github.io/}{https://unihm.github.io/}.