HapticCap: A Multimodal Dataset and Task for Understanding User Experience of Vibration Haptic Signals

TL;DR

This paper introduces HapticCap, a large multimodal dataset of vibration signals with user descriptions, and proposes a retrieval task to improve understanding and description of haptic signals using contrastive learning.

Contribution

The paper presents the first fully human-annotated haptic-captioned dataset and a novel retrieval task to enhance modeling of vibration signals with textual descriptions.

Findings

HapticCap contains 92,070 haptic-text pairs.

Language and audio models improve retrieval performance.

Separate training for description categories yields better results.

Abstract

Haptic signals, from smartphone vibrations to virtual reality touch feedback, can effectively convey information and enhance realism, but designing signals that resonate meaningfully with users is challenging. To facilitate this, we introduce a multimodal dataset and task, of matching user descriptions to vibration haptic signals, and highlight two primary challenges: (1) lack of large haptic vibration datasets annotated with textual descriptions as collecting haptic descriptions is time-consuming, and (2) limited capability of existing tasks and models to describe vibration signals in text. To advance this area, we create HapticCap, the first fully human-annotated haptic-captioned dataset, containing 92,070 haptic-text pairs for user descriptions of sensory, emotional, and associative attributes of vibrations. Based on HapticCap, we propose the haptic-caption retrieval task and present the results of this task from a supervised contrastive learning framework that brings together text representations within specific categories and vibrations. Overall, the combination of language model T5 and audio model AST yields the best performance in the haptic-caption retrieval task, especially when separately trained for each description category.