Controllable Music Production with Diffusion Models and Guidance Gradients

TL;DR

This paper introduces a method for controllable music generation using diffusion models with guidance, enabling tasks like inpainting, style transfer, and seamless transitions in high-quality stereo audio.

Contribution

It presents a flexible guidance framework at sampling time that supports various music production tasks with diffusion models, enhancing controllability and quality.

Findings

Effective inpainting, continuation, and style transfer of musical audio

Supports smooth transitions between different music tracks

Generates high-quality stereo audio at 44.1kHz

Abstract

We demonstrate how conditional generation from diffusion models can be used to tackle a variety of realistic tasks in the production of music in 44.1kHz stereo audio with sampling-time guidance. The scenarios we consider include continuation, inpainting and regeneration of musical audio, the creation of smooth transitions between two different music tracks, and the transfer of desired stylistic characteristics to existing audio clips. We achieve this by applying guidance at sampling time in a simple framework that supports both reconstruction and classification losses, or any combination of the two. This approach ensures that generated audio can match its surrounding context, or conform to a class distribution or latent representation specified relative to any suitable pre-trained classifier or embedding model. Audio samples are available at https://machinelearning.apple.com/research/controllable-music