Musical Composition Style Transfer via Disentangled Timbre Representations

TL;DR

This paper introduces deep learning models that disentangle pitch and timbre in musical audio, enabling style transfer and rearrangement of music across genres, validated through instrument detection and style transfer experiments.

Contribution

First deep learning approach for rearranging music of arbitrary genres by disentangling pitch and timbre, allowing style transfer without altering musical content.

Findings

Effective disentanglement of pitch and timbre achieved

Models successfully perform composition style transfer

Open-sourced code for further research

Abstract

Music creation involves not only composing the different parts (e.g., melody, chords) of a musical work but also arranging/selecting the instruments to play the different parts. While the former has received increasing attention, the latter has not been much investigated. This paper presents, to the best of our knowledge, the first deep learning models for rearranging music of arbitrary genres. Specifically, we build encoders and decoders that take a piece of polyphonic musical audio as input and predict as output its musical score. We investigate disentanglement techniques such as adversarial training to separate latent factors that are related to the musical content (pitch) of different parts of the piece, and that are related to the instrumentation (timbre) of the parts per short-time segment. By disentangling pitch and timbre, our models have an idea of how each piece was composed and arranged. Moreover, the models can realize "composition style transfer" by rearranging a musical piece without much affecting its pitch content. We validate the effectiveness of the models by experiments on instrument activity detection and composition style transfer. To facilitate follow-up research, we open source our code at https://github.com/biboamy/instrument-disentangle.