HapticLLaMA: A Multimodal Sensory Language Model for Haptic Captioning

TL;DR

HapticLLaMA is a novel multimodal language model that interprets vibration signals into descriptive language, advancing haptic captioning for virtual reality, accessibility, and rehabilitation, with improved human-aligned performance.

Contribution

The paper introduces HapticLLaMA, the first large language model designed for haptic captioning, integrating novel haptic tokenizers and reinforcement learning for better sensory understanding.

Findings

Achieved METEOR score of 59.98 and BLEU-4 score of 32.06.

Over 61% of captions rated above 3.5 by humans.

RLHF improved human rating distribution by 10%.

Abstract

Haptic captioning is the task of generating natural language descriptions from haptic signals, such as vibrations, for use in virtual reality, accessibility, and rehabilitation applications. While previous multimodal research has focused primarily on vision and audio, haptic signals for the sense of touch remain underexplored. To address this gap, we formalize the haptic captioning task and propose HapticLLaMA, a multimodal sensory language model that interprets vibration signals into descriptions in a given sensory, emotional, or associative category. We investigate two types of haptic tokenizers, a frequency-based tokenizer and an EnCodec-based tokenizer, that convert haptic signals into sequences of discrete units, enabling their integration with the LLaMA model. HapticLLaMA is trained in two stages: (1) supervised fine-tuning using the LLaMA architecture with LoRA-based adaptation, and (2) fine-tuning via reinforcement learning from human feedback (RLHF). We assess HapticLLaMA's captioning performance using both automated n-gram metrics and human evaluation. HapticLLaMA demonstrates strong capability in interpreting haptic vibration signals, achieving a METEOR score of 59.98 and a BLEU-4 score of 32.06 respectively. Additionally, over 61% of the generated captions received human ratings above 3.5 on a 7-point scale, with RLHF yielding a 10% improvement in the overall rating distribution, indicating stronger alignment with human haptic perception. These findings highlight the potential of large language models to process and adapt to sensory data.