LatBot: Distilling Universal Latent Actions for Vision-Language-Action Models

TL;DR

This paper introduces LatBot, a framework for learning universal latent actions from videos that improve robot manipulation by capturing physical priors and enabling effective transfer to new tasks with minimal data.

Contribution

It proposes a novel universal latent action learning method that incorporates action predictions and decomposes actions into motion and scene tokens for better transferability.

Findings

Achieves strong performance on simulated and real-world tasks.

Successfully completes five tasks with only 10 real-world trajectories.

Demonstrates effective few-shot transfer in robotic manipulation.

Abstract

Learning transferable latent actions from large-scale object manipulation videos can significantly enhance generalization in downstream robotics tasks, as such representations are agnostic to different robot embodiments. Existing approaches primarily rely on visual reconstruction objectives while neglecting physical priors, leading to sub-optimal performance in learning universal representations. To address these challenges, we propose a Universal Latent Action Learning framework that takes task instructions and multiple frames as inputs, and optimizes both future frame reconstruction and action sequence prediction. Unlike prior works, incorporating action predictions (e.g., gripper or hand trajectories and orientations) allows the model to capture richer physical priors such as real-world distances and orientations, thereby enabling seamless transferability to downstream tasks. We further decompose the latent actions into learnable motion and scene tokens to distinguish the robot's active movements from environmental changes, thus filtering out irrelevant dynamics. By distilling the learned latent actions into the latest VLA models, we achieve strong performance across both simulated (SIMPLER and LIBERO) and real-world robot settings. Notably, with only 10 real-world trajectories per task collected on a Franka robot, our approach successfully completes all five challenging tasks, demonstrating strong few-shot transferability in robotic manipulation.