ProSpect: Prompt Spectrum for Attribute-Aware Personalization of Diffusion Models

TL;DR

ProSpect introduces a novel method leveraging diffusion model stages and an expanded textual space to improve attribute-aware image personalization, enabling detailed editing without model fine-tuning.

Contribution

It proposes the Prompt Spectrum Space P* and a new image representation called ProSpect, enhancing disentanglement and controllability in personalized diffusion-based image generation.

Findings

Better disentanglement and controllability compared to existing methods.

Effective attribute-aware image editing without fine-tuning diffusion models.

Achieves high-quality, personalized image manipulations from a single input.

Abstract

Personalizing generative models offers a way to guide image generation with user-provided references. Current personalization methods can invert an object or concept into the textual conditioning space and compose new natural sentences for text-to-image diffusion models. However, representing and editing specific visual attributes such as material, style, and layout remains a challenge, leading to a lack of disentanglement and editability. To address this problem, we propose a novel approach that leverages the step-by-step generation process of diffusion models, which generate images from low to high frequency information, providing a new perspective on representing, generating, and editing images. We develop the Prompt Spectrum Space P*, an expanded textual conditioning space, and a new image representation method called \sysname. ProSpect represents an image as a collection of inverted textual token embeddings encoded from per-stage prompts, where each prompt corresponds to a specific generation stage (i.e., a group of consecutive steps) of the diffusion model. Experimental results demonstrate that P* and ProSpect offer better disentanglement and controllability compared to existing methods. We apply ProSpect in various personalized attribute-aware image generation applications, such as image-guided or text-driven manipulations of materials, style, and layout, achieving previously unattainable results from a single image input without fine-tuning the diffusion models. Our source code is available athttps://github.com/zyxElsa/ProSpect.