Per-Query Visual Concept Learning

TL;DR

This paper introduces a novel per-query personalization method for text-to-image models that enhances concept learning by leveraging attention-based loss terms and PDM features, leading to significant improvements over existing methods.

Contribution

The paper proposes a per-query personalization approach that improves visual concept learning by using attention-based losses and PDM features, enhancing existing methods.

Findings

Significant performance improvements over previous personalization techniques.

Effective across multiple models and personalization methods.

Enhanced semantic similarity for personalized concepts.

Abstract

Visual concept learning, also known as Text-to-image personalization, is the process of teaching new concepts to a pretrained model. This has numerous applications from product placement to entertainment and personalized design. Here we show that many existing methods can be substantially augmented by adding a personalization step that is (1) specific to the prompt and noise seed, and (2) using two loss terms based on the self- and cross- attention, capturing the identity of the personalized concept. Specifically, we leverage PDM features -- previously designed to capture identity -- and show how they can be used to improve personalized semantic similarity. We evaluate the benefit that our method gains on top of six different personalization methods, and several base text-to-image models (both UNet- and DiT-based). We find significant improvements even over previous per-query personalization methods.