Scaling Cross-Embodiment World Models for Dexterous Manipulation

TL;DR

This paper develops a unified world model for diverse robot embodiments, enabling transfer and generalization across different morphologies in dexterous manipulation tasks.

Contribution

It introduces a novel embodiment-invariant world model using particle-based representations, facilitating cross-embodiment learning and control transfer.

Findings

Scaling to more embodiments improves generalization.

Co-training on simulated and real data enhances performance.

Models enable effective control on various robot morphologies.

Abstract

Cross-embodiment learning seeks to build generalist robots that operate across diverse morphologies, but differences in action spaces and kinematics hinder data sharing and policy transfer. This raises a central question: Is there any invariance that allows actions to transfer across embodiments? We conjecture that environment dynamics are embodiment-invariant, and that world models capturing these dynamics can provide a unified interface across embodiments. To learn such a unified world model, the crucial step is to design state and action representations that abstract away embodiment-specific details while preserving control relevance. To this end, we represent different embodiments (e.g., human hands and robot hands) as sets of 3D particles and define actions as particle displacements, creating a shared representation for heterogeneous data and control problems. A graph-based world model is then trained on exploration data from diverse simulated robot hands and real human hands, and integrated with model-based planning for deployment on novel hardware. Experiments on rigid and deformable manipulation tasks reveal three findings: (i) scaling to more training embodiments improves generalization to unseen ones, (ii) co-training on both simulated and real data outperforms training on either alone, and (iii) the learned models enable effective control on robots with varied degrees of freedom. These results establish world models as a promising interface for cross-embodiment dexterous manipulation.