Latent Mappings: Generating Open-Ended Expressive Mappings Using Variational Autoencoders

TL;DR

This paper introduces latent mappings using Variational Autoencoders to generate expressive, open-ended gestural mappings for artistic and creative applications, demonstrated through a system mapping body movement to sound.

Contribution

It presents a novel approach leveraging VAE latent spaces to create open-ended, expressive mappings without requiring predefined input/output pairs.

Findings

Successfully generated expressive mappings in a dance context

Enabled creators to explore novel gestural mappings

Demonstrated system's potential for artistic expression

Abstract

In many contexts, creating mappings for gestural interactions can form part of an artistic process. Creators seeking a mapping that is expressive, novel, and affords them a sense of authorship may not know how to program it up in a signal processing patch. Tools like Wekinator and MIMIC allow creators to use supervised machine learning to learn mappings from example input/output pairings. However, a creator may know a good mapping when they encounter it yet start with little sense of what the inputs or outputs should be. We call this an open-ended mapping process. Addressing this need, we introduce the latent mapping, which leverages the latent space of an unsupervised machine learning algorithm such as a Variational Autoencoder trained on a corpus of unlabelled gestural data from the creator. We illustrate it with Sonified Body, a system mapping full-body movement to sound which we explore in a residency with three dancers.