Nested Attention: Semantic-aware Attention Values for Concept Personalization

TL;DR

This paper introduces Nested Attention, a novel mechanism for text-to-image personalization that enhances identity preservation and prompt alignment by injecting rich, query-dependent subject representations into cross-attention layers.

Contribution

The paper proposes Nested Attention, a new method that improves personalized image generation by integrating expressive, query-dependent subject features into existing models.

Findings

Enables high identity preservation in generated images.

Maintains strong alignment with input text prompts.

Allows combining multiple subjects from different domains.

Abstract

Personalizing text-to-image models to generate images of specific subjects across diverse scenes and styles is a rapidly advancing field. Current approaches often face challenges in maintaining a balance between identity preservation and alignment with the input text prompt. Some methods rely on a single textual token to represent a subject, which limits expressiveness, while others employ richer representations but disrupt the model's prior, diminishing prompt alignment. In this work, we introduce Nested Attention, a novel mechanism that injects a rich and expressive image representation into the model's existing cross-attention layers. Our key idea is to generate query-dependent subject values, derived from nested attention layers that learn to select relevant subject features for each region in the generated image. We integrate these nested layers into an encoder-based personalization method, and show that they enable high identity preservation while adhering to input text prompts. Our approach is general and can be trained on various domains. Additionally, its prior preservation allows us to combine multiple personalized subjects from different domains in a single image.