Listen to Your Map: An Online Representation for Spatial Sonification

TL;DR

This paper presents an online spatial sonification method that converts 3D scene geometry into auditory signals to aid navigation for visually impaired individuals, using depth sensors and a novel scene representation.

Contribution

It introduces a new mapping representation for scene geometry that enables accurate, online spatial sonification aligned with human auditory perception, improving navigation aid technology.

Findings

Enhanced accuracy and coverage in spatial sonification.

Effective handling of dynamic objects in the environment.

Improved timing and suitability for real-time navigation aid.

Abstract

Robotic perception is becoming a key technology for navigation aids, especially helping individuals with visual impairments through spatial sonification. This paper introduces a mapping representation that accurately captures scene geometry for sonification, turning physical spaces into auditory experiences. Using depth sensors, we encode an incrementally built 3D scene into a compact 360-degree representation with angular and distance information, aligning this way with human auditory spatial perception. The proposed framework performs localisation and mapping via VDB-Gaussian Process Distance Fields for efficient online scene reconstruction. The key aspect is a sensor-centric structure that maintains either a 2D-circular or 3D-cylindrical raster-based projection. This spatial representation is then converted into binaural auditory signals using simple pre-recorded responses from a representative room. Quantitative and qualitative evaluations show improvements in accuracy, coverage, timing and suitability for sonification compared to other approaches, with effective handling of dynamic objects as well. An accompanying video demonstrates spatial sonification in room-like environments. https://tinyurl.com/ListenToYourMap