VLA-Touch: Enhancing Vision-Language-Action Models with Dual-Level Tactile Feedback

TL;DR

VLA-Touch integrates tactile feedback into vision-language-action models using a pretrained tactile-language model and a diffusion-based controller, improving contact-rich task performance without fine-tuning the base model.

Contribution

The paper introduces a novel approach to incorporate tactile sensing into VLA models without fine-tuning, using a pretrained tactile-language model and a diffusion-based controller.

Findings

Enhanced task planning efficiency in real-world experiments.

Improved execution precision in contact-rich manipulation tasks.

Effective dual-level tactile feedback integration without fine-tuning.

Abstract

Tactile feedback is generally recognized to be crucial for effective interaction with the physical world. However, state-of-the-art Vision-Language-Action (VLA) models lack the ability to interpret and use tactile signals, limiting their effectiveness in contact-rich tasks. Incorporating tactile feedback into these systems is challenging due to the absence of large multi-modal datasets. We present VLA-Touch, an approach that enhances generalist robot policies with tactile sensing \emph{without fine-tuning} the base VLA. Our method introduces two key innovations: (1) a pipeline that leverages a pretrained tactile-language model that provides semantic tactile feedback for high-level task planning, and (2) a diffusion-based controller that refines VLA-generated actions with tactile signals for contact-rich manipulation. Through real-world experiments, we demonstrate that our dual-level integration of tactile feedback improves task planning efficiency while enhancing execution precision. Code is open-sourced at \href{https://github.com/jxbi1010/VLA-Touch}{this URL}.