SonoHaptics: An Audio-Haptic Cursor for Gaze-Based Object Selection in XR

TL;DR

SonoHaptics introduces an innovative audio-haptic cursor that leverages cross-modal mappings to improve gaze-based object selection in XR environments lacking reliable visual feedback, enhancing accuracy and user experience.

Contribution

The paper presents data-driven models and a computational approach for automatically generating cross-modal audio-haptic feedback based on visual object features in XR.

Findings

Enables accurate object identification without visual cues

Provides global and local feedback for scene differentiation

Improves gaze-based selection accuracy in cluttered scenes

Abstract

We introduce SonoHaptics, an audio-haptic cursor for gaze-based 3D object selection. SonoHaptics addresses challenges around providing accurate visual feedback during gaze-based selection in Extended Reality (XR), e.g., lack of world-locked displays in no- or limited-display smart glasses and visual inconsistencies. To enable users to distinguish objects without visual feedback, SonoHaptics employs the concept of cross-modal correspondence in human perception to map visual features of objects (color, size, position, material) to audio-haptic properties (pitch, amplitude, direction, timbre). We contribute data-driven models for determining cross-modal mappings of visual features to audio and haptic features, and a computational approach to automatically generate audio-haptic feedback for objects in the user's environment. SonoHaptics provides global feedback that is unique to each object in the scene, and local feedback to amplify differences between nearby objects. Our comparative evaluation shows that SonoHaptics enables accurate object identification and selection in a cluttered scene without visual feedback.