Subject-Diffusion:Open Domain Personalized Text-to-Image Generation without Test-time Fine-tuning

TL;DR

Subject-Diffusion is a new open-domain personalized text-to-image generation model that produces high-fidelity images of one or multiple subjects without test-time fine-tuning, using only a single reference image.

Contribution

It introduces a unified framework combining text and image semantics with attention control for multi-subject generation, and constructs a large-scale dataset for training.

Findings

Outperforms SOTA methods in single and multi-subject generation

Supports personalized generation without test-time fine-tuning

Effective in diverse open-domain scenarios

Abstract

Recent progress in personalized image generation using diffusion models has been significant. However, development in the area of open-domain and non-fine-tuning personalized image generation is proceeding rather slowly. In this paper, we propose Subject-Diffusion, a novel open-domain personalized image generation model that, in addition to not requiring test-time fine-tuning, also only requires a single reference image to support personalized generation of single- or multi-subject in any domain. Firstly, we construct an automatic data labeling tool and use the LAION-Aesthetics dataset to construct a large-scale dataset consisting of 76M images and their corresponding subject detection bounding boxes, segmentation masks and text descriptions. Secondly, we design a new unified framework that combines text and image semantics by incorporating coarse location and fine-grained reference image control to maximize subject fidelity and generalization. Furthermore, we also adopt an attention control mechanism to support multi-subject generation. Extensive qualitative and quantitative results demonstrate that our method outperforms other SOTA frameworks in single, multiple, and human customized image generation. Please refer to our \href{https://oppo-mente-lab.github.io/subject_diffusion/}{project page}