DreamBlend: Advancing Personalized Fine-tuning of Text-to-Image Diffusion Models

TL;DR

DreamBlend introduces a novel inference technique that combines early and late checkpoint features to enhance personalized text-to-image generation, balancing prompt fidelity, subject fidelity, and diversity.

Contribution

The paper proposes DreamBlend, a method that merges features from different fine-tuning checkpoints during inference to improve image quality and diversity.

Findings

Outperforms state-of-the-art fine-tuning methods in personalized image generation.

Achieves better balance of prompt fidelity, subject fidelity, and diversity.

Enables high-quality image synthesis on challenging prompts.

Abstract

Given a small number of images of a subject, personalized image generation techniques can fine-tune large pre-trained text-to-image diffusion models to generate images of the subject in novel contexts, conditioned on text prompts. In doing so, a trade-off is made between prompt fidelity, subject fidelity and diversity. As the pre-trained model is fine-tuned, earlier checkpoints synthesize images with low subject fidelity but high prompt fidelity and diversity. In contrast, later checkpoints generate images with low prompt fidelity and diversity but high subject fidelity. This inherent trade-off limits the prompt fidelity, subject fidelity and diversity of generated images. In this work, we propose DreamBlend to combine the prompt fidelity from earlier checkpoints and the subject fidelity from later checkpoints during inference. We perform a cross attention guided image synthesis from a later checkpoint, guided by an image generated by an earlier checkpoint, for the same prompt. This enables generation of images with better subject fidelity, prompt fidelity and diversity on challenging prompts, outperforming state-of-the-art fine-tuning methods.