SegQuant: A Semantics-Aware and Generalizable Quantization Framework for Diffusion Models

TL;DR

SegQuant is a versatile, semantics-aware quantization framework that significantly reduces the computational demands of diffusion models, facilitating deployment in resource-limited environments without retraining.

Contribution

It introduces a unified, adaptive quantization approach combining structural and dual-scale techniques, enhancing cross-model applicability and preserving output quality.

Findings

Achieves high accuracy with low-bit quantization across various diffusion models.

Maintains visual fidelity and model performance post-quantization.

Compatible with mainstream deployment tools and architectures.

Abstract

Diffusion models have demonstrated exceptional generative capabilities but are computationally intensive, posing significant challenges for deployment in resource-constrained or latency-sensitive environments. Quantization offers an effective means to reduce model size and computational cost, with post-training quantization (PTQ) being particularly appealing due to its compatibility with pre-trained models without requiring retraining or training data. However, existing PTQ methods for diffusion models often rely on architecture-specific heuristics that limit their generalizability and hinder integration with industrial deployment pipelines. To address these limitations, we propose SegQuant, a unified quantization framework that adaptively combines complementary techniques to enhance cross-model versatility. SegQuant consists of a segment-aware, graph-based quantization strategy (SegLinear) that captures structural semantics and spatial heterogeneity, along with a dual-scale quantization scheme (DualScale) that preserves polarity-asymmetric activations, which is crucial for maintaining visual fidelity in generated outputs. SegQuant is broadly applicable beyond Transformer-based diffusion models, achieving strong performance while ensuring seamless compatibility with mainstream deployment tools.