Diff-A-Riff: Musical Accompaniment Co-creation via Latent Diffusion Models

TL;DR

Diff-A-Riff is a Latent Diffusion Model that generates high-quality, adaptable musical accompaniments with efficient computation, controllable by audio or text, addressing limitations of previous deep generative music systems.

Contribution

The paper introduces Diff-A-Riff, a novel latent diffusion model for music accompaniment that improves audio quality, control flexibility, and computational efficiency over existing methods.

Findings

Produces 48kHz pseudo-stereo audio with high fidelity

Reduces inference time and memory usage significantly

Effective control via audio references and text prompts

Abstract

Recent advancements in deep generative models present new opportunities for music production but also pose challenges, such as high computational demands and limited audio quality. Moreover, current systems frequently rely solely on text input and typically focus on producing complete musical pieces, which is incompatible with existing workflows in music production. To address these issues, we introduce "Diff-A-Riff," a Latent Diffusion Model designed to generate high-quality instrumental accompaniments adaptable to any musical context. This model offers control through either audio references, text prompts, or both, and produces 48kHz pseudo-stereo audio while significantly reducing inference time and memory usage. We demonstrate the model's capabilities through objective metrics and subjective listening tests, with extensive examples available on the accompanying website: sonycslparis.github.io/diffariff-companion/