Tactile Modality Fusion for Vision-Language-Action Models

TL;DR

This paper introduces TacFiLM, a lightweight method for integrating tactile signals into vision-language-action models, enhancing robot manipulation performance without significant computational overhead.

Contribution

The paper presents TacFiLM, a novel post-training finetuning approach that conditions visual features on tactile data using FiLM, improving contact-rich manipulation tasks.

Findings

Improved success rate in insertion tasks

Enhanced force stability during manipulation

Faster completion times across tasks

Abstract

We propose TacFiLM, a lightweight modality-fusion approach that integrates visual-tactile signals into vision-language-action (VLA) models. While recent advances in VLA models have introduced robot policies that are both generalizable and semantically grounded, these models mainly rely on vision-based perception. Vision alone, however, cannot capture the complex interaction dynamics that occur during contact-rich manipulation, including contact forces, surface friction, compliance, and shear. While recent attempts to integrate tactile signals into VLA models often increase complexity through token concatenation or large-scale pretraining, the heavy computational demands of behavioural models necessitate more lightweight fusion strategies. To address these challenges, TacFiLM outlines a post-training finetuning approach that conditions intermediate visual features on pretrained tactile representations using feature-wise linear modulation (FiLM). Experimental results on insertion tasks demonstrate consistent improvements in success rate, direct insertion performance, completion time, and force stability across both in-distribution and out-of-distribution tasks. Together, these results support our method as an effective approach to integrating tactile signals into VLA models, improving contact-rich manipulation behaviours.