From Pixels to Tokens: A Systematic Study of Latent Action Supervision for Vision-Language-Action Models

TL;DR

This paper systematically compares different latent action supervision strategies in vision-language-action models, revealing their strengths in reasoning, generalization, and motor coordination, and demonstrating the effectiveness of discrete latent action tokens.

Contribution

It provides a unified framework for latent action supervision, compares four strategies, and offers insights into their respective advantages and optimal applications.

Findings

Image-based latent actions improve long-horizon reasoning.

Action-based latent actions enhance complex motor coordination.

Supervising with discrete latent action tokens yields the best performance.

Abstract

Latent actions serve as an intermediate representation that enables consistent modeling of vision-language-action (VLA) models across heterogeneous datasets. However, approaches to supervising VLAs with latent actions are fragmented and lack a systematic comparison. This work structures the study of latent action supervision from two perspectives: (i) regularizing the trajectory via image-based latent actions, and (ii) unifying the target space with action-based latent actions. Under a unified VLA baseline, we instantiate and compare four representative integration strategies. Our results reveal a formulation-task correspondence: image-based latent actions benefit long-horizon reasoning and scene-level generalization, whereas action-based latent actions excel at complex motor coordination. Furthermore, we find that directly supervising the VLM with discrete latent action tokens yields the most effective performance. Finally, our experiments offer initial insights into the benefits of latent action supervision in mixed-data, suggesting a promising direction for VLA training. Code is available at https://github.com/RUCKBReasoning/From_Pixels_to_Tokens.