VAT: Vision Action Transformer by Unlocking Full Representation of ViT

TL;DR

The paper introduces VAT, a novel Vision Transformer architecture that utilizes the full feature hierarchy for improved robotic perception and manipulation, achieving state-of-the-art results on LIBERO benchmarks.

Contribution

VAT extends ViT to process features across all layers, enabling deep perception-action fusion and improving robotic manipulation performance.

Findings

VAT achieves 98.15% success rate on LIBERO benchmarks.

VAT outperforms prior methods like OpenVLA-OFT.

Leveraging full feature hierarchy enhances robotic policy learning.

Abstract

In robot learning, Vision Transformers (ViTs) are standard for visual perception, yet most methods discard valuable information by using only the final layer's features. We argue this provides an insufficient representation and propose the Vision Action Transformer (VAT), a novel architecture that is extended from ViT and unlocks the full feature hierarchy of ViT. VAT processes specialized action tokens with visual features across all transformer layers, enabling a deep and progressive fusion of perception and action generation. On a suite of simulated manipulation tasks, VAT achieves a 98.15\% average success rate across four LIBERO benchmarks, establishing a new state-of-the-art by outperforming prior methods like OpenVLA-OFT. Our work presents not only a powerful model for imitation learning but also demonstrates the critical importance of leveraging the complete ''representation trajectory'' of vision models to advance robotic policy. The GitHub URL for the project code is https://github.com/sellerbubble/VAT.