UP-VLA: A Unified Understanding and Prediction Model for Embodied Agent

TL;DR

UP-VLA is a unified model that enhances embodied agent understanding and prediction by combining semantic comprehension with spatial reasoning, leading to significant improvements in benchmark and real-world tasks.

Contribution

The paper introduces UP-VLA, a novel training paradigm that jointly optimizes understanding and prediction for embodied agents, addressing limitations of existing VLA models.

Findings

33% improvement on Calvin ABC-D benchmark

Enhanced success in real-world spatial manipulation tasks

Better low-level spatial understanding in embodied control

Abstract

Recent advancements in Vision-Language-Action (VLA) models have leveraged pre-trained Vision-Language Models (VLMs) to improve the generalization capabilities. VLMs, typically pre-trained on vision-language understanding tasks, provide rich semantic knowledge and reasoning abilities. However, prior research has shown that VLMs often focus on high-level semantic content and neglect low-level features, limiting their ability to capture detailed spatial information and understand physical dynamics. These aspects, which are crucial for embodied control tasks, remain underexplored in existing pre-training paradigms. In this paper, we investigate the training paradigm for VLAs, and introduce \textbf{UP-VLA}, a \textbf{U}nified VLA model training with both multi-modal \textbf{U}nderstanding and future \textbf{P}rediction objectives, enhancing both high-level semantic comprehension and low-level spatial understanding. Experimental results show that UP-VLA achieves a 33% improvement on the Calvin ABC-D benchmark compared to the previous state-of-the-art method. Additionally, UP-VLA demonstrates improved success rates in real-world manipulation tasks, particularly those requiring precise spatial information.