Temporal Feature Matters: A Framework for Diffusion Model Quantization

TL;DR

This paper presents a novel quantization framework for diffusion models that preserves temporal features, reduces disturbances, and accelerates image generation, addressing the limitations of existing post-training quantization methods.

Contribution

The paper introduces a new quantization framework with three strategies—TIB-based maintenance, cache-based maintenance, and disturbance-aware selection—to improve diffusion model compression and efficiency.

Findings

Outperforms existing methods in preserving temporal features.

Achieves significant acceleration in diffusion model inference.

Maintains high-quality image generation after quantization.

Abstract

The Diffusion models, widely used for image generation, face significant challenges related to their broad applicability due to prolonged inference times and high memory demands. Efficient Post-Training Quantization (PTQ) is crucial to address these issues. However, unlike traditional models, diffusion models critically rely on the time-step for the multi-round denoising. Typically, each time-step is encoded into a hypersensitive temporal feature by several modules. Despite this, existing PTQ methods do not optimize these modules individually. Instead, they employ unsuitable reconstruction objectives and complex calibration methods, leading to significant disturbances in the temporal feature and denoising trajectory, as well as reduced compression efficiency. To address these challenges, we introduce a novel quantization framework that includes three strategies: 1) TIB-based Maintenance: Based on our innovative Temporal Information Block (TIB) definition, Temporal Information-aware Reconstruction (TIAR) and Finite Set Calibration (FSC) are developed to efficiently align original temporal features. 2) Cache-based Maintenance: Instead of indirect and complex optimization for the related modules, pre-computing and caching quantized counterparts of temporal features are developed to minimize errors. 3) Disturbance-aware Selection: Employ temporal feature errors to guide a fine-grained selection between the two maintenance strategies for further disturbance reduction. This framework preserves most of the temporal information and ensures high-quality end-to-end generation. Extensive testing on various datasets, diffusion models and hardware confirms our superior performance and acceleration.