A Data Perspective on Enhanced Identity Preservation for Diffusion Personalization

TL;DR

This paper introduces a data-centric regularization strategy for personalized text-to-image models, significantly improving identity preservation of specific subjects like logos and pets without modifying model architecture.

Contribution

A novel data generation approach that enhances identity preservation in text-to-image models, applicable across different architectures, setting new state-of-the-art results.

Findings

Improved identity preservation on benchmarks

Enhanced text alignment accuracy

Applicable to various model architectures

Abstract

Large text-to-image models have revolutionized the ability to generate imagery using natural language. However, particularly unique or personal visual concepts, such as pets and furniture, will not be captured by the original model. This has led to interest in how to personalize a text-to-image model. Despite significant progress, this task remains a formidable challenge, particularly in preserving the subject's identity. Most researchers attempt to address this issue by modifying model architectures. These methods are capable of keeping the subject structure and color but fail to preserve identity details. Towards this issue, our approach takes a data-centric perspective. We introduce a novel regularization dataset generation strategy on both the text and image level. This strategy enables the model to preserve fine details of the desired subjects, such as text and logos. Our method is architecture-agnostic and can be flexibly applied on various text-to-image models. We show on established benchmarks that our data-centric approach forms the new state of the art in terms of identity preservation and text alignment.