DiffPro: Joint Timestep and Layer-Wise Precision Optimization for Efficient Diffusion Inference

TL;DR

DiffPro is a post-training framework that jointly optimizes timestep count and layer precision in diffusion models, significantly reducing inference latency and memory usage without retraining.

Contribution

It introduces a unified approach combining sensitivity metrics, dynamic quantization, and timestep selection to enhance diffusion inference efficiency.

Findings

Achieves up to 6.25x model compression

Reduces timesteps by 50%

Speeds up inference by 2.8x with Delta FID <= 10

Abstract

Diffusion models produce high quality images but inference is costly due to many denoising steps and heavy matrix operations. We present DiffPro, a post-training, hardware-faithful framework that works with the exact integer kernels used in deployment and jointly tunes timesteps and per-layer precision in Diffusion Transformers (DiTs) to reduce latency and memory without any training. DiffPro combines three parts: a manifold-aware sensitivity metric to allocate weight bits, dynamic activation quantization to stabilize activations across timesteps, and a budgeted timestep selector guided by teacher-student drift. In experiments DiffPro achieves up to 6.25x model compression, fifty percent fewer timesteps, and 2.8x faster inference with Delta FID <= 10 on standard benchmarks, demonstrating practical efficiency gains. DiffPro unifies step reduction and precision planning into a single budgeted deployable plan for real-time energy-aware diffusion inference.