Sound2Hap: Learning Audio-to-Vibrotactile Haptic Generation from Human Ratings

TL;DR

This paper introduces Sound2Hap, a CNN-based model that generates meaningful vibrations from environmental sounds, outperforming traditional methods in perceptual harmony and broadening sound-driven haptic applications.

Contribution

It presents a data-driven, perceptually validated model for audio-to-vibration translation that generalizes across diverse environmental sounds, unlike prior signal-processing approaches.

Findings

Sound2Hap outperforms baseline algorithms in perceptual ratings.

Participants rated Sound2Hap's vibrations as more harmonious with sounds.

The model demonstrates low latency and broad applicability.

Abstract

Environmental sounds like footsteps, keyboard typing, or dog barking carry rich information and emotional context, making them valuable for designing haptics in user applications. Existing audio-to-vibration methods, however, rely on signal-processing rules tuned for music or games and often fail to generalize across diverse sounds. To address this, we first investigated user perception of four existing audio-to-haptic algorithms, then created a data-driven model for environmental sounds. In Study 1, 34 participants rated vibrations generated by the four algorithms for 1,000 sounds, revealing no consistent algorithm preferences. Using this dataset, we trained Sound2Hap, a CNN-based autoencoder, to generate perceptually meaningful vibrations from diverse sounds with low latency. In Study 2, 15 participants rated its output higher than signal-processing baselines on both audio-vibration match and Haptic Experience Index (HXI), finding it more harmonious with diverse sounds. This work demonstrates a perceptually validated approach to audio-haptic translation, broadening the reach of sound-driven haptics.