Latent Action Diffusion for Cross-Embodiment Manipulation

TL;DR

This paper introduces a diffusion policy framework in a latent action space that unifies diverse robot end-effectors, enabling effective cross-embodiment manipulation and improving multi-robot skill transfer.

Contribution

It proposes a novel latent action space learned via contrastive encoders, facilitating cross-embodiment manipulation and data sharing across different robot morphologies.

Findings

Achieved up to 25.3% improvement in manipulation success rates.

Learned semantically aligned latent spaces for various robotic hands and grippers.

Enabled multi-robot control with a single unified policy.

Abstract

End-to-end learning is emerging as a powerful paradigm for robotic manipulation, but its effectiveness is limited by data scarcity and the heterogeneity of action spaces across robot embodiments. In particular, diverse action spaces across different end-effectors create barriers for cross-embodiment learning and skill transfer. We address this challenge through diffusion policies learned in a latent action space that unifies diverse end-effector actions. We first show that we can learn a semantically aligned latent action space for anthropomorphic robotic hands, a human hand, and a parallel jaw gripper using encoders trained with a contrastive loss. Second, we show that by using our proposed latent action space for co-training on manipulation data from different end-effectors, we can utilize a single policy for multi-robot control and obtain up to 25.3% improved manipulation success rates, indicating successful skill transfer despite a significant embodiment gap. Our approach using latent cross-embodiment policies presents a new method to unify different action spaces across embodiments, enabling efficient multi-robot control and data sharing across robot setups. This unified representation significantly reduces the need for extensive data collection for each new robot morphology, accelerates generalization across embodiments, and ultimately facilitates more scalable and efficient robotic learning.