UniLACT: Depth-Aware RGB Latent Action Learning for Vision-Language-Action Models

TL;DR

UniLACT introduces a depth-aware transformer-based model for vision-language-action tasks, enhancing latent action representations with 3D geometric structure for improved manipulation performance.

Contribution

It proposes UniLACT and UniLARN, novel frameworks for depth-aware latent action learning that incorporate geometric priors into vision-language-action models.

Findings

Outperforms RGB-only latent action models in simulation and real-world tasks.

Effectively incorporates depth information to improve spatial understanding.

Demonstrates robustness across in-domain and out-of-domain scenarios.

Abstract

Latent action representations learned from unlabeled videos have recently emerged as a promising paradigm for pretraining vision-language-action (VLA) models without explicit robot action supervision. However, latent actions derived solely from RGB observations primarily encode appearance-driven dynamics and lack explicit 3D geometric structure, which is essential for precise and contact-rich manipulation. To address this limitation, we introduce UniLACT, a transformer-based VLA model that incorporates geometric structure through depth-aware latent pretraining, enabling downstream policies to inherit stronger spatial priors. To facilitate this process, we propose UniLARN, a unified latent action learning framework based on inverse and forward dynamics objectives that learns a shared embedding space for RGB and depth while explicitly modeling their cross-modal interactions. This formulation produces modality-specific and unified latent action representations that serve as pseudo-labels for the depth-aware pretraining of UniLACT. Extensive experiments in both simulation and real-world settings demonstrate the effectiveness of depth-aware unified latent action representations. UniLACT consistently outperforms RGB-based latent action baselines under in-domain and out-of-domain pretraining regimes, as well as on both seen and unseen manipulation tasks.The project page is at https://manishgovind.github.io/unilact-vla/