World Models for Learning Dexterous Hand-Object Interactions from Human Videos

TL;DR

This paper introduces DexWM, a world model for dexterous hand-object interactions that predicts future states from fine-grained finger actions extracted from videos, enabling effective transfer to robotic manipulation tasks.

Contribution

DexWM is the first world model to incorporate finger keypoints and hand configuration consistency for modeling dexterous manipulation from human videos.

Findings

DexWM outperforms prior models in future-state prediction.

It achieves over 50% improvement in zero-shot transfer to robotic tasks.

It successfully models fine-grained dexterous interactions from video data.

Abstract

Modeling dexterous hand-object interactions is challenging as it requires understanding how subtle finger motions influence the environment through contact with objects. While recent world models address interaction modeling, they typically rely on coarse action spaces that fail to capture fine-grained dexterity. We, therefore, introduce DexWM, a Dexterous Interaction World Model that predicts future latent states of the environment conditioned on past states and dexterous actions. To overcome the scarcity of finely annotated dexterous datasets, DexWM represents actions using finger keypoints extracted from egocentric videos, enabling training on over 900 hours of human and non-dexterous robot data. Further, to accurately model dexterity, we find that predicting visual features alone is insufficient; therefore, we incorporate an auxiliary hand consistency loss that enforces accurate hand configurations. DexWM outperforms prior world models conditioned on text, navigation, or full-body actions in future-state prediction and demonstrates strong zero-shot transfer to unseen skills on a Franka Panda arm with an Allegro gripper, surpassing Diffusion Policy by over 50% on average across grasping, placing, and reaching tasks.