MagicNaming: Consistent Identity Generation by Finding a "Name Space" in T2I Diffusion Models

TL;DR

This paper introduces MagicNaming, a method to find a 'Name Space' in diffusion models' feature space, enabling consistent identity generation for faces by using name embeddings, without affecting the original text-to-image capabilities.

Contribution

The paper proposes a novel approach to identify a 'Name Space' in diffusion models, allowing for consistent identity generation through name embeddings, enhancing face generation control.

Findings

Name embeddings effectively ensure identity consistency in generated images.

The approach preserves the original text-to-image generation capabilities.

Identity-aware models can be created by integrating name embeddings into existing diffusion models.

Abstract

Large-scale text-to-image diffusion models, (e.g., DALL-E, SDXL) are capable of generating famous persons by simply referring to their names. Is it possible to make such models generate generic identities as simple as the famous ones, e.g., just use a name? In this paper, we explore the existence of a "Name Space", where any point in the space corresponds to a specific identity. Fortunately, we find some clues in the feature space spanned by text embedding of celebrities' names. Specifically, we first extract the embeddings of celebrities' names in the Laion5B dataset with the text encoder of diffusion models. Such embeddings are used as supervision to learn an encoder that can predict the name (actually an embedding) of a given face image. We experimentally find that such name embeddings work well in promising the generated image with good identity consistency. Note that like the names of celebrities, our predicted name embeddings are disentangled from the semantics of text inputs, making the original generation capability of text-to-image models well-preserved. Moreover, by simply plugging such name embeddings, all variants (e.g., from Civitai) derived from the same base model (i.e., SDXL) readily become identity-aware text-to-image models. Project homepage: \url{https://magicfusion.github.io/MagicNaming/}.