InstructVLA: Vision-Language-Action Instruction Tuning from Understanding to Manipulation

TL;DR

InstructVLA is an end-to-end vision-language-action model that combines multimodal reasoning with precise manipulation, achieving superior performance in complex tasks and generalization to new benchmarks through a novel training paradigm.

Contribution

It introduces VLA-IT, a new training method that jointly optimizes reasoning and action generation, bridging the gap between vision-language understanding and robotic manipulation.

Findings

33% improvement on SimplerEnv tasks

Outperforms fine-tuned OpenVLA by 96% on SimplerEnv-Instruct

Leverages textual reasoning to enhance real-world manipulation

Abstract

To operate effectively in the real world, robots should integrate multimodal reasoning with precise action generation. However, existing vision-language-action (VLA) models often sacrifice one for the other, narrow their abilities to task-specific manipulation data, and suffer catastrophic forgetting of pre-trained vision-language capabilities. To bridge this gap, we introduce InstructVLA, an end-to-end VLA model that preserves the flexible reasoning of large vision-language models (VLMs) while delivering leading manipulation performance with the help of embodied reasoning. InstructVLA introduces a novel training paradigm, Vision-Language-Action Instruction Tuning (VLA-IT), which employs multimodal training with mixture-of-experts adaptation to jointly optimize embodied reasoning and action generation on both standard VLM corpora and a curated 650K-sample VLA-IT dataset. On in-domain SimplerEnv tasks, InstructVLA achieves 33% improvement over SpatialVLA. To evaluate generalization, we introduce SimplerEnv-Instruct, an 80-task benchmark requiring closed-loop control and high-level instruction understanding, where it outperforms a fine-tuned OpenVLA by 96% and an action expert aided by GPT-4o by 29%. Additionally, InstructVLA surpasses baseline VLMs on multimodal tasks and exhibits inference-time scaling by leveraging textual reasoning to boost manipulation performance in both simulated and real-world settings. These results demonstrate InstructVLA's potential for bridging intuitive and steerable human-robot interaction with efficient policy learning.