Steering Guidance for Personalized Text-to-Image Diffusion Models

TL;DR

This paper introduces a novel personalization guidance method for text-to-image diffusion models that balances subject fidelity and text alignment by dynamically interpolating between pre-trained and fine-tuned models, improving image generation quality.

Contribution

The paper proposes a simple, effective guidance technique using an unlearned weak model and weight interpolation, addressing limitations of existing guidance methods in personalized diffusion models.

Findings

Improves text alignment and target distribution fidelity.

Seamlessly integrates with various fine-tuning strategies.

No additional computational overhead during inference.

Abstract

Personalizing text-to-image diffusion models is crucial for adapting the pre-trained models to specific target concepts, enabling diverse image generation. However, fine-tuning with few images introduces an inherent trade-off between aligning with the target distribution (e.g., subject fidelity) and preserving the broad knowledge of the original model (e.g., text editability). Existing sampling guidance methods, such as classifier-free guidance (CFG) and autoguidance (AG), fail to effectively guide the output toward well-balanced space: CFG restricts the adaptation to the target distribution, while AG compromises text alignment. To address these limitations, we propose personalization guidance, a simple yet effective method leveraging an unlearned weak model conditioned on a null text prompt. Moreover, our method dynamically controls the extent of unlearning in a weak model through weight interpolation between pre-trained and fine-tuned models during inference. Unlike existing guidance methods, which depend solely on guidance scales, our method explicitly steers the outputs toward a balanced latent space without additional computational overhead. Experimental results demonstrate that our proposed guidance can improve text alignment and target distribution fidelity, integrating seamlessly with various fine-tuning strategies.