Learning to Customize Text-to-Image Diffusion In Diverse Context

TL;DR

This paper introduces a simple, cost-effective method to improve text-to-image model customization by diversifying textual prompts, enhancing semantic alignment and prompt fidelity without architectural changes.

Contribution

The authors propose a novel approach that diversifies personal concept context solely in textual prompts, significantly boosting customization quality across multiple baseline methods.

Findings

Improved CLIP scores across different customization methods.

Enhanced semantic alignment in textual and image spaces.

Method is compatible with existing customization techniques.

Abstract

Most text-to-image customization techniques fine-tune models on a small set of \emph{personal concept} images captured in minimal contexts. This often results in the model becoming overfitted to these training images and unable to generalize to new contexts in future text prompts. Existing customization methods are built on the success of effectively representing personal concepts as textual embeddings. Thus, in this work, we resort to diversifying the context of these personal concepts \emph{solely} within the textual space by simply creating a contextually rich set of text prompts, together with a widely used self-supervised learning objective. Surprisingly, this straightforward and cost-effective method significantly improves semantic alignment in the textual space, and this effect further extends to the image space, resulting in higher prompt fidelity for generated images. Additionally, our approach does not require any architectural modifications, making it highly compatible with existing text-to-image customization methods. We demonstrate the broad applicability of our approach by combining it with four different baseline methods, achieving notable CLIP score improvements.