DECOR:Decomposition and Projection of Text Embeddings for Text-to-Image Customization

TL;DR

DECOR is a novel method that decomposes and projects text embeddings to reduce overfitting in text-to-image models, improving customization accuracy and fidelity to input prompts.

Contribution

The paper introduces DECOR, a technique that projects text embeddings onto a space orthogonal to undesired tokens, addressing overfitting in T2I customization.

Findings

DECOR outperforms state-of-the-art models in customization tasks.

DECOR achieves better text and visual alignment metrics.

Images generated by DECOR are more faithful to input prompts.

Abstract

Text-to-image (T2I) models can effectively capture the content or style of reference images to perform high-quality customization. A representative technique for this is fine-tuning using low-rank adaptations (LoRA), which enables efficient model customization with reference images. However, fine-tuning with a limited number of reference images often leads to overfitting, resulting in issues such as prompt misalignment or content leakage. These issues prevent the model from accurately following the input prompt or generating undesired objects during inference. To address this problem, we examine the text embeddings that guide the diffusion model during inference. This study decomposes the text embedding matrix and conducts a component analysis to understand the embedding space geometry and identify the cause of overfitting. Based on this, we propose DECOR, which projects text embeddings onto a vector space orthogonal to undesired token vectors, thereby reducing the influence of unwanted semantics in the text embeddings. Experimental results demonstrate that DECOR outperforms state-of-the-art customization models and achieves Pareto frontier performance across text and visual alignment evaluation metrics. Furthermore, it generates images more faithful to the input prompts, showcasing its effectiveness in addressing overfitting and enhancing text-to-image customization.