HapticLDM: A Diffusion Model for Text-to-Vibrotactile Generation

TL;DR

HapticLDM is a novel diffusion-based model that generates accurate, coherent vibrations from text, improving realism and semantic alignment for applications like metaverse, games, and film.

Contribution

It introduces the first diffusion model for text-to-vibration generation, with a new data strategy and global denoising mechanism for better dynamic and semantic modeling.

Findings

Outperforms state-of-the-art autoregressive models in realism and semantic accuracy.

Enhances haptic design workflow with diverse and physically precise vibrations.

User study shows improved user experience and satisfaction.

Abstract

Text-to-vibration generation converts natural language into haptic feedback, enabling vibration-effect designers to get scenarios-fitted vibrations more efficiently, which shows great potentials in application fields such as metaverse, games, and film to enrich the user experience in interactive scenarios. The core challenge in this field is how to generate accurate, consistent, and complete vibrations according to textual semantics. Very recent autoregressive (AR) approaches (e.g., HapticGen) exhibit limited capacity in fully capturing global dependencies, owing to the inherent sequential nature of their modeling and prevailing data constraints. In this paper, we proposed HapticLDM, the first text-to-vibration generative model built upon Latent Diffusion Models (LDMs). Firstly, with respect to the data, we introduced a text-processing strategy that emphasizes dynamic characteristics to curate high-quality data pairs for fine-grained dynamic modeling. Secondly, HapticLDM incorporates a global denoising mechanism that regulates coherent and stable variations in the temporal envelope. Furthermore, we conduct extensive evaluations, including A/B testing against the state-of-the-art baseline and a user study involving 30 participants. The results demonstrate that our model enhances realism and semantic alignment. Qualitative feedback further indicates that HapticLDM simplifies the haptic design workflow while generating diverse, subtle, and physically precise vibrations.