PTQ4ADM: Post-Training Quantization for Efficient Text Conditional Audio Diffusion Models

TL;DR

This paper presents PTQ4ADM, a post-training quantization framework for audio diffusion models that significantly reduces model size with minimal impact on synthesis quality, enabling efficient deployment in resource-limited settings.

Contribution

The paper introduces novel prompt augmentation and activation-aware calibration techniques for quantizing text-conditional audio diffusion models, maintaining high quality with reduced model size.

Findings

Model size reduced by up to 70%

Quantization to 4-bit weights and 8-bit activations preserves quality

Achieves comparable synthesis quality with full-precision models

Abstract

Denoising diffusion models have emerged as state-of-the-art in generative tasks across image, audio, and video domains, producing high-quality, diverse, and contextually relevant data. However, their broader adoption is limited by high computational costs and large memory footprints. Post-training quantization (PTQ) offers a promising approach to mitigate these challenges by reducing model complexity through low-bandwidth parameters. Yet, direct application of PTQ to diffusion models can degrade synthesis quality due to accumulated quantization noise across multiple denoising steps, particularly in conditional tasks like text-to-audio synthesis. This work introduces PTQ4ADM, a novel framework for quantizing audio diffusion models(ADMs). Our key contributions include (1) a coverage-driven prompt augmentation method and (2) an activation-aware calibration set generation algorithm for text-conditional ADMs. These techniques ensure comprehensive coverage of audio aspects and modalities while preserving synthesis fidelity. We validate our approach on TANGO, Make-An-Audio, and AudioLDM models for text-conditional audio generation. Extensive experiments demonstrate PTQ4ADM's capability to reduce the model size by up to 70\% while achieving synthesis quality metrics comparable to full-precision models($<$5\% increase in FD scores). We show that specific layers in the backbone network can be quantized to 4-bit weights and 8-bit activations without significant quality loss. This work paves the way for more efficient deployment of ADMs in resource-constrained environments.