Mugeetion: Musical Interface Using Facial Gesture and Emotion

TL;DR

Mugeetion is a novel musical interface that captures users' facial gestures to detect emotional states and dynamically modify sound, demonstrated through an interactive sound installation to explore emotion-sound connections.

Contribution

This work introduces a new interface translating facial gestures into musical features, enabling emotion-driven sound generation in real-time applications.

Findings

Successfully captured emotional states from facial gestures.

Enabled real-time emotion-based sound modulation.

Demonstrated in an interactive sound installation.

Abstract

People feel emotions when listening to music. However, emotions are not tangible objects that can be exploited in the music composition process as they are difficult to capture and quantify in algorithms. We present a novel musical interface, Mugeetion, designed to capture occurring instances of emotional states from users' facial gestures and relay that data to associated musical features. Mugeetion can translate qualitative data of emotional states into quantitative data, which can be utilized in the sound generation process. We also presented and tested this work in the exhibition of sound installation, Hearing Seascape, using the audiences' facial expressions. Audiences heard changes in the background sound based on their emotional state. The process contributes multiple research areas, such as gesture tracking systems, emotion-sound modeling, and the connection between sound and facial gesture.