Gradient-Free Classifier Guidance for Diffusion Model Sampling

TL;DR

This paper introduces Gradient-free Classifier Guidance (GFCG), an efficient method that improves image fidelity and class prediction accuracy in diffusion models by utilizing a pre-trained classifier without gradient descent.

Contribution

The paper proposes a novel gradient-free guidance technique for diffusion models that enhances image quality and class accuracy without additional computational cost.

Findings

GFCG improves class prediction accuracy in diffusion sampling.

Combining GFCG with Autoguidance enhances image fidelity and diversity.

Achieved a new record FD_DINOv2 of 23.09 on ImageNet 512x512.

Abstract

Image generation using diffusion models have demonstrated outstanding learning capabilities, effectively capturing the full distribution of the training dataset. They are known to generate wide variations in sampled images, albeit with a trade-off in image fidelity. Guided sampling methods, such as classifier guidance (CG) and classifier-free guidance (CFG), focus sampling in well-learned high-probability regions to generate images of high fidelity, but each has its limitations. CG is computationally expensive due to the use of back-propagation for classifier gradient descent, while CFG, being gradient-free, is more efficient but compromises class label alignment compared to CG. In this work, we propose an efficient guidance method that fully utilizes a pre-trained classifier without using gradient descent. By using the classifier solely in inference mode, a time-adaptive reference class label and corresponding guidance scale are determined at each time step for guided sampling. Experiments on both class-conditioned and text-to-image generation diffusion models demonstrate that the proposed Gradient-free Classifier Guidance (GFCG) method consistently improves class prediction accuracy. We also show GFCG to be complementary to other guided sampling methods like CFG. When combined with the state-of-the-art Autoguidance (ATG), without additional computational overhead, it enhances image fidelity while preserving diversity. For ImageNet 512$\times$512, we achieve a record $\text{FD}_{\text{DINOv2}}$ of 23.09, while simultaneously attaining a higher classification Precision (94.3%) compared to ATG (90.2%)