MUSTEM: A Dual-Modality System for Vibrotactile and Visual Translation of Music as an Assistive Technology

TL;DR

MUSTEM is a dual-modality system that translates music into vibrotactile and visual formats, enhancing musical accessibility for the deaf community through real-time audio analysis and intuitive visual interfaces.

Contribution

The paper introduces a novel multisensorial system combining tactile and visual translation of music, with open-source hardware and software components for accessibility.

Findings

Preliminary positive feedback from deaf users on vibrotactile perception.

System effectively maps musical features to tactile and visual outputs.

Open-source release facilitates further development and adoption.

Abstract

The emotional and structural experience of music remains a significant accessibility challenge for the deaf and hard of hearing community. This paper introduces MUSTEM (Multisensorial Emotional Translation), a novel system designed to translate music into a rich, coherent, and scientifically-grounded sensory experience. We present a dual-modality approach addressing this challenge through two interconnected components. First, a low-cost, portable hardware prototype that performs real-time audio analysis, mapping distinct frequency bands (sub-bass, bass, mid-range, treble) to a four-channel vibrotactile system, allowing users to feel the music's rhythmic and foundational structure. Second, to overcome the processing limitations of embedded hardware, we developed a high-fidelity software simulation that demonstrates the full potential of the visual translation. This assistive dashboard decodes musical components - such as rhythm, harmony, and frequency spectrum - into an intuitive and educational visual interface. MUSTEM offers a comprehensive framework for sensory substitution, presenting a viable and accessible pathway for the deaf community to experience music not just as vibration, but as a structured, substantiated and emotionally resonant visual and tactile language. Preliminary feedback from seven deaf users suggests the system's spatial vibrotactile mapping is perceptible and engaging. All source code and hardware designs are released as open-source. Video demonstrations and open-source code are available on the project's official channel.