DisMix: Disentangling Mixtures of Musical Instruments for Source-level Pitch and Timbre Manipulation

TL;DR

DisMix is a novel generative framework that disentangles pitch and timbre in multi-instrument music mixtures, enabling manipulation and synthesis of new instrument combinations and musical attributes.

Contribution

It introduces a modular approach for source-level pitch and timbre disentanglement in multi-instrument music, filling a gap in existing single-instrument focused methods.

Findings

Successfully disentangles pitch and timbre in multi-instrument mixtures

Enables manipulation of instrument attributes and creation of novel instrument combinations

Effective on both simple chords and complex chorale datasets

Abstract

Existing work on pitch and timbre disentanglement has been mostly focused on single-instrument music audio, excluding the cases where multiple instruments are presented. To fill the gap, we propose DisMix, a generative framework in which the pitch and timbre representations act as modular building blocks for constructing the melody and instrument of a source, and the collection of which forms a set of per-instrument latent representations underlying the observed mixture. By manipulating the representations, our model samples mixtures with novel combinations of pitch and timbre of the constituent instruments. We can jointly learn the disentangled pitch-timbre representations and a latent diffusion transformer that reconstructs the mixture conditioned on the set of source-level representations. We evaluate the model using both a simple dataset of isolated chords and a realistic four-part chorales in the style of J.S. Bach, identify the key components for the success of disentanglement, and demonstrate the application of mixture transformation based on source-level attribute manipulation.