Feelings, Not Feel: Affective Audio-Visual Pseudo-Haptics in Hand-Tracked XR

TL;DR

This study investigates how affective audio-visual pseudo-haptic cues on a virtual hand influence user feelings and perceptions in hand-tracked XR, revealing that such cues modulate emotional states rather than simulate tactile sensations.

Contribution

It demonstrates that pseudo-haptic cues in XR primarily serve as affective feedback rather than direct tactile substitutes, expanding understanding of multisensory integration in virtual environments.

Findings

Affective cues systematically shift user emotions

Participants rarely perceive sustained tactile sensations

Pseudo-haptics modulate arousal and valence in XR

Abstract

Hand-tracking enables controller-free XR interaction but does not have the tactile feedback controllers provide. Rather than treating this solely as a missing-sensation problem, we explore whether pseudo-haptic cues on an embodied virtual hand act as tactile or as affect substitutes that shape how interactions feel. We used a mixed reality prototype that keeps the contacted surface visually neutral, rendering cues on the hand with motion modulation for texture, color glow, and movement-coupled sound. In a within-subjects study (n=12), participants experienced 12 conditions (4 effects x 3 modalities: audio, visual, both) and reported subjective affect and cognitive demand. Participants rarely reported sustained tactile, thermal sensations, yet affect shifted systematically: rough-hot lowered valence increasing arousal, while smooth-cold produced calmer pleasant states. These findings suggest that pseudo-haptics in XR may be better understood as an affective feedback channel rather than a direct replacement for physical touch in controller-free systems.