Guiding Token-Sparse Diffusion Models

TL;DR

This paper introduces Sparse Guidance, a method that uses token-level sparsity to improve the inference performance of token-sparse diffusion models, achieving high-quality image synthesis with fewer computational resources.

Contribution

The paper proposes Sparse Guidance, a novel approach that enhances token-sparse diffusion models during inference by using token-level sparsity, leading to better quality and efficiency.

Findings

Achieves 1.58 FID on ImageNet-256 with 25% fewer FLOPs.

Yields up to 58% FLOP savings at matched quality.

Improves composition and human preference scores in large-scale models.

Abstract

Diffusion models deliver high quality in image synthesis but remain expensive during training and inference. Recent works have leveraged the inherent redundancy in visual content to make training more affordable by training only on a subset of visual information. While these methods were successful in providing cheaper and more effective training, sparsely trained diffusion models struggle in inference. This is due to their lacking response to Classifier-free Guidance (CFG) leading to underwhelming performance during inference. To overcome this, we propose Sparse Guidance (SG). Instead of using conditional dropout as a signal to guide diffusion models, SG uses token-level sparsity. As a result, SG preserves the high-variance of the conditional prediction better, achieving good quality and high variance outputs. Leveraging token-level sparsity at inference, SG improves fidelity at lower compute, achieving 1.58 FID on the commonly used ImageNet-256 benchmark with 25% fewer FLOPs, and yields up to 58% FLOP savings at matched baseline quality. To demonstrate the effectiveness of Sparse Guidance, we train a 2.5B text-to-image diffusion model using training time sparsity and leverage SG during inference. SG achieves improvements in composition and human preference score while increasing throughput at the same time.