How Do VLAs Effectively Inherit from VLMs?

TL;DR

This paper introduces GrinningFace, a diagnostic benchmark for evaluating how effectively vision-language-action models inherit knowledge from vision-language models, highlighting the importance of knowledge transfer techniques for embodied AI.

Contribution

The paper presents a novel emoji tabletop manipulation benchmark and systematically evaluates various knowledge transfer methods for VLA models.

Findings

Preserving VLM priors enhances generalization in embodied control.

Parameter-efficient fine-tuning improves knowledge transfer.

Emoji-based tasks reveal the effectiveness of different transfer techniques.

Abstract

Vision-language-action (VLA) models hold the promise to attain generalizable embodied control. To achieve this, a pervasive paradigm is to leverage the rich vision-semantic priors of large vision-language models (VLMs). However, the fundamental question persists: How do VLAs effectively inherit the prior knowledge from VLMs? To address this critical question, we introduce a diagnostic benchmark, GrinningFace, an emoji tabletop manipulation task where the robot arm is asked to place objects onto printed emojis corresponding to language instructions. This task design is particularly revealing -- knowledge associated with emojis is ubiquitous in Internet-scale datasets used for VLM pre-training, yet emojis themselves are largely absent from standard robotics datasets. Consequently, they provide a clean proxy: successful task completion indicates effective transfer of VLM priors to embodied control. We implement this diagnostic task in both simulated environment and a real robot, and compare various promising techniques for knowledge transfer. Specifically, we investigate the effects of parameter-efficient fine-tuning, VLM freezing, co-training, predicting discretized actions, and predicting latent actions. Through systematic evaluation, our work not only demonstrates the critical importance of preserving VLM priors for the generalization of VLA but also establishes guidelines for future research in developing truly generalizable embodied AI systems.