JEPA-VLA: Video Predictive Embedding is Needed for VLA Models

TL;DR

This paper demonstrates that incorporating video-based predictive embeddings significantly enhances vision-language-action models' efficiency and generalization in robotic manipulation tasks.

Contribution

Introducing JEPA-VLA, a method that integrates video predictive embeddings into VLAs, addressing their limitations in environment understanding and policy generalization.

Findings

JEPA-VLA improves performance on multiple benchmarks.

Predictive video embeddings better capture environment dynamics.

Enhanced sample efficiency and generalization in robotic tasks.

Abstract

Recent vision-language-action (VLA) models built upon pretrained vision-language models (VLMs) have achieved significant improvements in robotic manipulation. However, current VLAs still suffer from low sample efficiency and limited generalization. This paper argues that these limitations are closely tied to an overlooked component, pretrained visual representation, which offers insufficient knowledge on both aspects of environment understanding and policy prior. Through an in-depth analysis, we find that commonly used visual representations in VLAs, whether pretrained via language-image contrastive learning or image-based self-supervised learning, remain inadequate at capturing crucial, task-relevant environment information and at inducing effective policy priors, i.e., anticipatory knowledge of how the environment evolves under successful task execution. In contrast, we discover that predictive embeddings pretrained on videos, in particular V-JEPA 2, are adept at flexibly discarding unpredictable environment factors and encoding task-relevant temporal dynamics, thereby effectively compensating for key shortcomings of existing visual representations in VLAs. Building on these observations, we introduce JEPA-VLA, a simple yet effective approach that adaptively integrates predictive embeddings into existing VLAs. Our experiments demonstrate that JEPA-VLA yields substantial performance gains across a range of benchmarks, including LIBERO, LIBERO-plus, RoboTwin2.0, and real-robot tasks.