TMPDiff: Temporal Mixed-Precision for Diffusion Models

TL;DR

TMPDiff introduces a novel temporal mixed-precision approach for diffusion models, assigning different numeric precisions to denoising timesteps, which improves perceptual quality and efficiency over uniform-precision methods.

Contribution

It proposes an adaptive algorithm for per-step precision assignment in diffusion models, validated across multiple models and datasets, enhancing inference quality and speed.

Findings

Achieves 10-20% better perceptual quality at matched speedup.

On FLUX.1-dev, reaches 90% SSIM with 2.5x speedup.

Outperforms uniform-precision baselines consistently.

Abstract

Diffusion models are the go-to method for Text-to-Image generation, but their iterative denoising processes has high inference latency. Quantization reduces compute time by using lower bitwidths, but applies a fixed precision across all denoising timesteps, leaving an entire optimization axis unexplored. We propose TMPDiff, a temporal mixed-precision framework for diffusion models that assigns different numeric precision to different denoising timesteps. We hypothesize that quantization errors accumulate additively across timesteps, which we then validate experimentally. Based on our observations, we develop an adaptive bisectioning-based algorithm, which assigns per-step precisions with linear evaluation complexity, reducing an otherwise exponential search problem. Across four state-of-the-art diffusion models and three datasets, TMPDiff consistently outperforms uniform-precision baselines at matched speedup, achieving 10 to 20% improvement in perceptual quality. On FLUX.1-dev, TMPDiff achieves 90% SSIM relative to the full-precision model at a speedup of 2.5x over 16-bit inference.