Flexiffusion: Segment-wise Neural Architecture Search for Flexible Denoising Schedule

TL;DR

Flexiffusion introduces a training-free neural architecture search method that optimizes diffusion model generation steps and network structures, significantly accelerating image generation while maintaining quality.

Contribution

It proposes a novel flexible step segmentation and search paradigm for diffusion models, reducing search costs and enabling faster inference without additional training.

Findings

Achieved up to 5.1x speedup on Stable Diffusion V1.5

Reduced redundancy in diffusion models effectively

Validated on multiple datasets with positive results

Abstract

Diffusion models are cutting-edge generative models adept at producing diverse, high-quality images. Despite their effectiveness, these models often require significant computational resources owing to their numerous sequential denoising steps and the significant inference cost of each step. Recently, Neural Architecture Search (NAS) techniques have been employed to automatically search for faster generation processes. However, NAS for diffusion is inherently time-consuming as it requires estimating thousands of diffusion models to search for the optimal one. In this paper, we introduce Flexiffusion, a novel training-free NAS paradigm designed to accelerate diffusion models by concurrently optimizing generation steps and network structures. Specifically, we partition the generation process into isometric step segments, each sequentially composed of a full step, multiple partial steps, and several null steps. The full step computes all network blocks, while the partial step involves part of the blocks, and the null step entails no computation. Flexiffusion autonomously explores flexible step combinations for each segment, substantially reducing search costs and enabling greater acceleration compared to the state-of-the-art (SOTA) method for diffusion models. Our searched models reported speedup factors of $2.6\times$ and $1.5\times$ for the original LDM-4-G and the SOTA, respectively. The factors for Stable Diffusion V1.5 and the SOTA are $5.1\times$ and $2.0\times$. We also verified the performance of Flexiffusion on multiple datasets, and positive experiment results indicate that Flexiffusion can effectively reduce redundancy in diffusion models.