Classifier-free Guidance with Adaptive Scaling

TL;DR

This paper introduces $eta$-CFG, an adaptive guidance method for diffusion models that dynamically balances image quality and prompt fidelity, improving FID scores while maintaining CLIP similarity.

Contribution

The paper proposes $eta$-CFG, a novel adaptive guidance technique that stabilizes and dynamically adjusts guidance strength during diffusion, enhancing image quality and prompt alignment.

Findings

Improved FID scores over standard CFG.

Maintained CLIP similarity comparable to reference CFG.

Effective dynamic adjustment of guidance during diffusion.

Abstract

Classifier-free guidance (CFG) is an essential mechanism in contemporary text-driven diffusion models. In practice, in controlling the impact of guidance we can see the trade-off between the quality of the generated images and correspondence to the prompt. When we use strong guidance, generated images fit the conditioned text perfectly but at the cost of their quality. Dually, we can use small guidance to generate high-quality results, but the generated images do not suit our prompt. In this paper, we present $\beta$-CFG ($\beta$-adaptive scaling in Classifier-Free Guidance), which controls the impact of guidance during generation to solve the above trade-off. First, $\beta$-CFG stabilizes the effects of guiding by gradient-based adaptive normalization. Second, $\beta$-CFG uses the family of single-modal ($\beta$-distribution), time-dependent curves to dynamically adapt the trade-off between prompt matching and the quality of samples during the diffusion denoising process. Our model obtained better FID scores, maintaining the text-to-image CLIP similarity scores at a level similar to that of the reference CFG.