Audio-VLA: Adding Contact Audio Perception to Vision-Language-Action Model for Robotic Manipulation

TL;DR

Audio-VLA enhances robotic manipulation by integrating contact audio perception with vision-language-action models, enabling better understanding of dynamic processes and contact events, validated through extensive simulation and real-world experiments.

Contribution

This work introduces Audio-VLA, a multimodal manipulation policy that incorporates contact audio into VLA models and proposes the TCR metric for evaluating dynamic process perception.

Findings

Audio-VLA outperforms vision-only models in manipulation tasks.

The TCR metric effectively measures dynamic process perception.

Multimodal integration improves robotic understanding of contact events.

Abstract

The Vision-Language-Action models (VLA) have achieved significant advances in robotic manipulation recently. However, vision-only VLA models create fundamental limitations, particularly in perceiving interactive and manipulation dynamic processes. This paper proposes Audio-VLA, a multimodal manipulation policy that leverages contact audio to perceive contact events and dynamic process feedback. Audio-VLA overcomes the vision-only constraints of VLA models. Additionally, this paper introduces the Task Completion Rate (TCR) metric to systematically evaluate dynamic operational processes. Audio-VLA employs pre-trained DINOv2 and SigLIP as visual encoders, AudioCLIP as the audio encoder, and Llama2 as the large language model backbone. We apply LoRA fine-tuning to these pre-trained modules to achieve robust cross-modal understanding of both visual and acoustic inputs. A multimodal projection layer aligns features from different modalities into the same feature space. Moreover RLBench and LIBERO simulation environments are enhanced by adding collision-based audio generation to provide realistic sound feedback during object interactions. Since current robotic manipulation evaluations focus on final outcomes rather than providing systematic assessment of dynamic operational processes, the proposed TCR metric measures how well robots perceive dynamic processes during manipulation, creating a more comprehensive evaluation metric. Extensive experiments on LIBERO, RLBench, and two real-world tasks demonstrate Audio-VLA's superior performance over vision-only comparative methods, while the TCR metric effectively quantifies dynamic process perception capabilities.