The Language of Touch: Translating Vibrations into Text with Dual-Branch Learning

TL;DR

This paper introduces ViPAC, a novel dual-branch learning method for translating vibrotactile signals into natural language descriptions, addressing the unique properties of vibrotactile data.

Contribution

It proposes a new approach for vibrotactile captioning that handles hybrid periodic-aperiodic signals and introduces a new dataset, LMT108-CAP, for this task.

Findings

ViPAC outperforms baseline methods in lexical fidelity.

The dual-branch strategy effectively disentangles signal components.

The new dataset enables better evaluation of vibrotactile captioning.

Abstract

The standardization of vibrotactile data by IEEE P1918.1 workgroup has greatly advanced its applications in virtual reality, human-computer interaction and embodied artificial intelligence. Despite these efforts, the semantic interpretation and understanding of vibrotactile signals remain an unresolved challenge. In this paper, we make the first attempt to address vibrotactile captioning, {\it i.e.}, generating natural language descriptions from vibrotactile signals. We propose Vibrotactile Periodic-Aperiodic Captioning (ViPAC), a method designed to handle the intrinsic properties of vibrotactile data, including hybrid periodic-aperiodic structures and the lack of spatial semantics. Specifically, ViPAC employs a dual-branch strategy to disentangle periodic and aperiodic components, combined with a dynamic fusion mechanism that adaptively integrates signal features. It also introduces an orthogonality constraint and weighting regularization to ensure feature complementarity and fusion consistency. Additionally, we construct LMT108-CAP, the first vibrotactile-text paired dataset, using GPT-4o to generate five constrained captions per surface image from the popular LMT-108 dataset. Experiments show that ViPAC significantly outperforms the baseline methods adapted from audio and image captioning, achieving superior lexical fidelity and semantic alignment.