D$^2$-DPM: Dual Denoising for Quantized Diffusion Probabilistic Models

TL;DR

This paper introduces D2-DPM, a dual denoising approach that effectively mitigates quantization noise in diffusion models, enabling faster, compressed image generation with improved quality.

Contribution

The paper proposes a novel dual denoising mechanism to counteract quantization noise effects in diffusion models, enhancing efficiency and maintaining high generation quality.

Findings

Achieves 1.42 lower FID than full-precision models

Realizes 3.99x model compression

Attains 11.67x acceleration in bit operations

Abstract

Diffusion models have achieved cutting-edge performance in image generation. However, their lengthy denoising process and computationally intensive score estimation network impede their scalability in low-latency and resource-constrained scenarios. Post-training quantization (PTQ) compresses and accelerates diffusion models without retraining, but it inevitably introduces additional quantization noise, resulting in mean and variance deviations. In this work, we propose D2-DPM, a dual denoising mechanism aimed at precisely mitigating the adverse effects of quantization noise on the noise estimation network. Specifically, we first unravel the impact of quantization noise on the sampling equation into two components: the mean deviation and the variance deviation. The mean deviation alters the drift coefficient of the sampling equation, influencing the trajectory trend, while the variance deviation magnifies the diffusion coefficient, impacting the convergence of the sampling trajectory. The proposed D2-DPM is thus devised to denoise the quantization noise at each time step, and then denoise the noisy sample through the inverse diffusion iterations. Experimental results demonstrate that D2-DPM achieves superior generation quality, yielding a 1.42 lower FID than the full-precision model while achieving 3.99x compression and 11.67x bit-operation acceleration.