MR4MR: Mixed Reality for Melody Reincarnation

TL;DR

This paper presents MR4MR, a mixed reality sound installation enabling users to interact with virtual objects that generate and transform melodies based on their physical environment, blending real-world interaction with AI-driven music generation.

Contribution

It introduces a novel mixed reality system that combines user interaction with machine learning-based music generation to create dynamic ambient melodies.

Findings

Users can interactively generate and modify melodies in real-time.

The system successfully blends physical interactions with AI-driven music evolution.

Enhanced immersive musical experiences in mixed reality environments.

Abstract

There is a long history of an effort made to explore musical elements with the entities and spaces around us, such as musique concr\`ete and ambient music. In the context of computer music and digital art, interactive experiences that concentrate on the surrounding objects and physical spaces have also been designed. In recent years, with the development and popularization of devices, an increasing number of works have been designed in Extended Reality to create such musical experiences. In this paper, we describe MR4MR, a sound installation work that allows users to experience melodies produced from interactions with their surrounding space in the context of Mixed Reality (MR). Using HoloLens, an MR head-mounted display, users can bump virtual objects that emit sound against real objects in their surroundings. Then, by continuously creating a melody following the sound made by the object and re-generating randomly and gradually changing melody using music generation machine learning models, users can feel their ambient melody "reincarnating".