PixelVLA: Advancing Pixel-level Understanding in Vision-Language-Action Model

TL;DR

PixelVLA introduces a novel vision-language-action model that enhances pixel-level scene understanding and multimodal prompting, significantly improving robot manipulation success rates with reduced training costs.

Contribution

It is the first VLA model supporting pixel-level reasoning and multimodal prompts, built on a new instruction tuning framework and a large-scale pixel annotation dataset.

Findings

Improves manipulation success rates by up to 28.7%.

Requires only 1.5% of the pretraining cost of previous models.

Demonstrates enhanced accuracy and versatility in complex environments.

Abstract

Vision-Language-Action models (VLAs) are emerging as powerful tools for learning generalizable visuomotor control policies. However, current VLAs are mostly trained on large-scale image-text-action data and remain limited in two key ways: (i) they struggle with pixel-level scene understanding, and (ii) they rely heavily on textual prompts, which reduces their flexibility in real-world settings. To address these challenges, we introduce PixelVLA, the first VLA model designed to support both pixel-level reasoning and multimodal prompting with text and visual inputs. Our approach is built on a new visuomotor instruction tuning framework that integrates a multiscale pixel-aware encoder with a visual promptaware encoder. To train PixelVLA effectively, we further propose a two-stage automated annotation pipeline that generates Pixel-160K, a large-scale dataset with pixel-level annotations derived from existing robot data. Experiments on three standard VLA benchmarks and two VLA model variants show that PixelVLA improves manipulation success rates by 10.1%-28.7% over OpenVLA, while requiring only 1.5% of its pretraining cost. These results demonstrate that PixelVLA can be integrated into existing VLAs to enable more accurate, efficient, and versatile robot control in complex environments.