Interpretable Timbre Synthesis using Variational Autoencoders Regularized on Timbre Descriptors

TL;DR

This paper introduces a regularized VAE model for timbre synthesis that incorporates timbre descriptors and harmonic content to improve interpretability and control over sound generation.

Contribution

It proposes a novel regularization technique for VAEs that integrates timbre descriptors and harmonic content for more interpretable and concise timbre synthesis.

Findings

Enhanced interpretability of latent space

Reduced dimensionality of sound representation

Improved control over timbre synthesis

Abstract

Controllable timbre synthesis has been a subject of research for several decades, and deep neural networks have been the most successful in this area. Deep generative models such as Variational Autoencoders (VAEs) have the ability to generate a high-level representation of audio while providing a structured latent space. Despite their advantages, the interpretability of these latent spaces in terms of human perception is often limited. To address this limitation and enhance the control over timbre generation, we propose a regularized VAE-based latent space that incorporates timbre descriptors. Moreover, we suggest a more concise representation of sound by utilizing its harmonic content, in order to minimize the dimensionality of the latent space.