Vector-Quantized Timbre Representation

TL;DR

This paper introduces a vector-quantized auto-encoder for flexible, disentangled timbre synthesis and transfer, enabling high-quality audio translation between instruments and voice, with potential for descriptor-based sound synthesis.

Contribution

It proposes a novel discrete latent space auto-encoder that disentangles timbre from loudness for improved sound synthesis and transfer capabilities.

Findings

Effective timbre transfer between instruments and voice.

Disentangled spectral features enable flexible synthesis.

Mapping latent space to acoustic descriptors facilitates descriptor-based synthesis.

Abstract

Timbre is a set of perceptual attributes that identifies different types of sound sources. Although its definition is usually elusive, it can be seen from a signal processing viewpoint as all the spectral features that are perceived independently from pitch and loudness. Some works have studied high-level timbre synthesis by analyzing the feature relationships of different instruments, but acoustic properties remain entangled and generation bound to individual sounds. This paper targets a more flexible synthesis of an individual timbre by learning an approximate decomposition of its spectral properties with a set of generative features. We introduce an auto-encoder with a discrete latent space that is disentangled from loudness in order to learn a quantized representation of a given timbre distribution. Timbre transfer can be performed by encoding any variable-length input signals into the quantized latent features that are decoded according to the learned timbre. We detail results for translating audio between orchestral instruments and singing voice, as well as transfers from vocal imitations to instruments as an intuitive modality to drive sound synthesis. Furthermore, we can map the discrete latent space to acoustic descriptors and directly perform descriptor-based synthesis.