Pioneering 4-Bit FP Quantization for Diffusion Models: Mixup-Sign Quantization and Timestep-Aware Fine-Tuning

TL;DR

This paper introduces a novel 4-bit floating-point quantization method for diffusion models, using mixup-sign quantization and timestep-aware fine-tuning to improve efficiency and performance over existing integer-based approaches.

Contribution

The paper proposes the MSFP framework with unsigned FP quantization, TALoRA, and DFA, addressing key challenges and achieving superior 4-bit FP quantization performance for diffusion models.

Findings

First to achieve superior 4-bit FP quantization performance.

Outperforms existing PTQ fine-tuning methods in 4-bit INT quantization.

Demonstrates improved memory efficiency and inference speed.

Abstract

Model quantization reduces the bit-width of weights and activations, improving memory efficiency and inference speed in diffusion models. However, achieving 4-bit quantization remains challenging. Existing methods, primarily based on integer quantization and post-training quantization fine-tuning, struggle with inconsistent performance. Inspired by the success of floating-point (FP) quantization in large language models, we explore low-bit FP quantization for diffusion models and identify key challenges: the failure of signed FP quantization to handle asymmetric activation distributions, the insufficient consideration of temporal complexity in the denoising process during fine-tuning, and the misalignment between fine-tuning loss and quantization error. To address these challenges, we propose the mixup-sign floating-point quantization (MSFP) framework, first introducing unsigned FP quantization in model quantization, along with timestep-aware LoRA (TALoRA) and denoising-factor loss alignment (DFA), which ensure precise and stable fine-tuning. Extensive experiments show that we are the first to achieve superior performance in 4-bit FP quantization for diffusion models, outperforming existing PTQ fine-tuning methods in 4-bit INT quantization.