What Matters in Building Vision-Language-Action Models for Generalist Robots

TL;DR

This paper investigates key design factors for Vision-Language-Action models in robotics, introduces RoboVLMs with minimal manual design, and provides extensive experimental insights and an open-source framework for future research.

Contribution

It identifies crucial design choices for VLAs, develops a new flexible family of RoboVLMs achieving state-of-the-art results, and offers comprehensive experimental guidance and open-source tools.

Findings

VLA performance depends on backbone, architecture, and data integration choices.

RoboVLMs achieve new state-of-the-art in simulation and real-world tasks.

Extensive experiments with over 8 backbones and 600 configurations provide detailed design insights.

Abstract

To utilize Foundation Vision Language Models (VLMs) for robotic tasks and motion planning, the community has proposed different methods for injecting action components into VLMs and building the Vision-Language-Action models (VLAs). In this work, we disclose the key factors that significantly influence the performance of VLA on robot manipulation problems and focus on answering three essential design choices: which backbone to select, how to formulate the VLA architectures, and when to add cross-embodiment data. The obtained results convince us firmly to explain why we prefer VLA and develop a new family of VLAs, RoboVLMs, which require very few manual designs and achieve a new state-of-the-art performance in three simulation tasks and real-world experiments. Through our extensive experiments, which include over 8 VLM backbones, 4 policy architectures, and over 600 distinct designed experiments, we provide a detailed guidebook for the future design of VLAs. In addition to the study, the highly flexible RoboVLMs framework, which supports easy integrations of new VLMs and free combinations of various design choices, is made public to facilitate future research. We open-source all details, including codes, models, datasets, and toolkits, along with detailed training and evaluation recipes at: robovlms.github.io.