Improving Diffusion Generalization with Weak-to-Strong Segmented Guidance

TL;DR

This paper introduces a hybrid guidance method called SGG based on the weak-to-strong principle, improving diffusion model generalization during sampling and training, validated through extensive experiments.

Contribution

The paper proposes SGG, a novel hybrid guidance technique for diffusion models, and demonstrates its effectiveness in both inference and training settings, enhancing generalization.

Findings

SGG outperforms existing guidance methods at inference.

Training with W2S principle improves diffusion model generalization.

SGG achieves performance gains in both conditional and unconditional tasks.

Abstract

Diffusion models generate synthetic images through an iterative refinement process. However, the misalignment between the simulation-free objective and the iterative process often causes accumulated gradient error along the sampling trajectory, which leads to unsatisfactory results and a failure to generalize. Guidance techniques like Classifier Free Guidance (CFG) and AutoGuidance (AG) alleviate this by extrapolating between the main and inferior signal for stronger generalization. Despite empirical success, the effective operational regimes of prevalent guidance methods are still under-explored, leading to ambiguity when selecting the appropriate guidance method given a precondition. In this work, we first conduct synthetic comparisons to isolate and demonstrate the effective regime of guidance methods represented by CFG and AG from the perspective of weak-to-strong principle. Based on this, we propose a hybrid instantiation called SGG under the principle, taking the benefits of both. Furthermore, we demonstrate that the W2S principle along with SGG can be migrated into the training objective, improving the generalization ability of unguided diffusion models. We validate our approach with comprehensive experiments. At inference time, evaluations on SD3 and SD3.5 confirm that SGG outperforms existing training-free guidance variants. Training-time experiments on transformer architectures demonstrate the effective migration and performance gains in both conditional and unconditional settings. Code is available at https://github.com/851695e35/SGG.