Learning a Latent Space of Multitrack Measures

TL;DR

This paper extends MusicVAE to create a latent space for multitrack measures, enabling meaningful music generation, interpolation, and attribute manipulation with chord conditioning for harmony control.

Contribution

It introduces a novel latent space representation for multitrack music with chord conditioning, allowing flexible and controlled music generation and manipulation.

Findings

Enables generation of plausible multitrack measures from scratch

Allows interpolation between musical measures in a meaningful way

Supports manipulation of musical attributes while maintaining harmony

Abstract

Discovering and exploring the underlying structure of multi-instrumental music using learning-based approaches remains an open problem. We extend the recent MusicVAE model to represent multitrack polyphonic measures as vectors in a latent space. Our approach enables several useful operations such as generating plausible measures from scratch, interpolating between measures in a musically meaningful way, and manipulating specific musical attributes. We also introduce chord conditioning, which allows all of these operations to be performed while keeping harmony fixed, and allows chords to be changed while maintaining musical "style". By generating a sequence of measures over a predefined chord progression, our model can produce music with convincing long-term structure. We demonstrate that our latent space model makes it possible to intuitively control and generate musical sequences with rich instrumentation (see https://goo.gl/s2N7dV for generated audio).