GeoVLA: Empowering 3D Representations in Vision-Language-Action Models

TL;DR

GeoVLA introduces a 3D-aware vision-language-action framework for robotics, significantly improving spatial understanding and manipulation capabilities by integrating 3D geometric data with language and visual inputs.

Contribution

It is the first to effectively incorporate 3D geometric information into VLA models, enhancing robotic manipulation and spatial awareness.

Findings

Achieves state-of-the-art results on LIBERO and ManiSkill2 benchmarks.

Demonstrates robustness in real-world tasks with height and scale variations.

Outperforms existing models in simulation and real-world environments.

Abstract

Vision-Language-Action (VLA) models have emerged as a promising approach for enabling robots to follow language instructions and predict corresponding actions. However, current VLA models mainly rely on 2D visual inputs, neglecting the rich geometric information in the 3D physical world, which limits their spatial awareness and adaptability. In this paper, we present GeoVLA, a novel VLA framework that effectively integrates 3D information to advance robotic manipulation. It uses a vision-language model (VLM) to process images and language instructions,extracting fused vision-language embeddings. In parallel, it converts depth maps into point clouds and employs a customized point encoder, called Point Embedding Network, to generate 3D geometric embeddings independently. These produced embeddings are then concatenated and processed by our proposed spatial-aware action expert, called 3D-enhanced Action Expert, which combines information from different sensor modalities to produce precise action sequences. Through extensive experiments in both simulation and real-world environments, GeoVLA demonstrates superior performance and robustness. It achieves state-of-the-art results in the LIBERO and ManiSkill2 simulation benchmarks and shows remarkable robustness in real-world tasks requiring height adaptability, scale awareness and viewpoint invariance.