Audio Generation Through Score-Based Generative Modeling: Design Principles and Implementation

TL;DR

This paper reviews diffusion-based generative models for audio, emphasizing design principles, implementation guidance, and providing an open-source framework demonstrated through diverse audio applications.

Contribution

It offers a comprehensive review of diffusion model design principles for audio, introduces an open-source codebase, and evaluates applications like audio generation and speech synthesis.

Findings

Effective diffusion model configurations for audio quality

Open-source framework facilitates reproducibility and rapid prototyping

Benchmark results demonstrate model versatility across tasks

Abstract

Diffusion models have emerged as powerful deep generative techniques, producing high-quality and diverse samples in applications in various domains including audio. While existing reviews provide overviews, there remains limited in-depth discussion of these specific design choices. The audio diffusion model literature also lacks principled guidance for the implementation of these design choices and their comparisons for different applications. This survey provides a comprehensive review of diffusion model design with an emphasis on design principles for quality improvement and conditioning for audio applications. We adopt the score modeling perspective as a unifying framework that accommodates various interpretations, including recent approaches like flow matching. We systematically examine the training and sampling procedures of diffusion models, and audio applications through different conditioning mechanisms. To provide an integrated, unified codebase and to promote reproducible research and rapid prototyping, we introduce an open-source codebase (https://github.com/gzhu06/AudioDiffuser) that implements our reviewed framework for various audio applications. We demonstrate its capabilities through three case studies: audio generation, speech enhancement, and text-to-speech synthesis, with benchmark evaluations on standard datasets.