Identity Encoder for Personalized Diffusion

TL;DR

This paper introduces an encoder-based method for personalized image generation using diffusion models, enabling efficient identity representation extraction from few images without extensive fine-tuning.

Contribution

It proposes a novel identity encoder that allows personalized image generation with minimal data, reducing computational and storage costs compared to traditional fine-tuning methods.

Findings

Outperforms fine-tuning approaches in image quality and reconstruction.

Achieves over 95% user preference in evaluations.

Works effectively with few reference images for new identities.

Abstract

Many applications can benefit from personalized image generation models, including image enhancement, video conferences, just to name a few. Existing works achieved personalization by fine-tuning one model for each person. While being successful, this approach incurs additional computation and storage overhead for each new identity. Furthermore, it usually expects tens or hundreds of examples per identity to achieve the best performance. To overcome these challenges, we propose an encoder-based approach for personalization. We learn an identity encoder which can extract an identity representation from a set of reference images of a subject, together with a diffusion generator that can generate new images of the subject conditioned on the identity representation. Once being trained, the model can be used to generate images of arbitrary identities given a few examples even if the model hasn't been trained on the identity. Our approach greatly reduces the overhead for personalized image generation and is more applicable in many potential applications. Empirical results show that our approach consistently outperforms existing fine-tuning based approach in both image generation and reconstruction, and the outputs is preferred by users more than 95% of the time compared with the best performing baseline.