LACE: Latent Visual Representation for Cross-Embodiment Learning

TL;DR

LACE is a framework that aligns human and robot visual representations in a shared latent space, enabling effective cross-embodiment learning and zero-shot transfer of robot policies using minimal demonstration data.

Contribution

LACE introduces a novel semantic alignment method leveraging shared body part correspondences and self-supervised learning to improve cross-embodiment robot learning.

Findings

Zero-shot transfer policies using LACE-DINO outperform baseline by 65%.

LACE improves performance in low-data and out-of-distribution environments.

Single robot demonstration suffices for training the model.

Abstract

Cross-embodiment learning from human demonstrations is hindered by the visual gap between human and robot embodiments. While self-supervised learning (SSL) backbones encode rich inter-class semantics of general objects, we show they fail to establish correspondence between human and robot hands. We propose LACE, a framework that aligns human and robot visual representations in the latent space of these backbones by leveraging correspondences between shared body parts across embodiments as sparse supervision. These annotations can be automatically obtained via forward kinematics, and single robot demonstration is sufficient to train the model. Our semantic alignment loss matches distributions incurred by corresponding features, lifting patch-level supervision to semantic-level alignment, while a Gram loss preserves pretrained feature quality. This alignment enables robot policies to leverage abundant human data when robot demonstrations are scarce: in zero-shot transfer, policies using LACE-DINO outperform those using DINO by a large margin (65\%), with consistent gains in low-data regimes and out-of-distribution environments.