Multi-Concept Customization of Text-to-Image Diffusion

TL;DR

This paper introduces Custom Diffusion, a quick and efficient method to teach text-to-image models new concepts using minimal tuning, enabling the composition of multiple concepts in generated images.

Contribution

It presents a novel approach to rapidly customize text-to-image models for new concepts by optimizing only a few parameters, allowing for fast training and flexible concept composition.

Findings

Achieves concept learning in approximately 6 minutes.

Enables seamless combination of multiple concepts in generated images.

Outperforms or matches existing methods in quality and efficiency.

Abstract

While generative models produce high-quality images of concepts learned from a large-scale database, a user often wishes to synthesize instantiations of their own concepts (for example, their family, pets, or items). Can we teach a model to quickly acquire a new concept, given a few examples? Furthermore, can we compose multiple new concepts together? We propose Custom Diffusion, an efficient method for augmenting existing text-to-image models. We find that only optimizing a few parameters in the text-to-image conditioning mechanism is sufficiently powerful to represent new concepts while enabling fast tuning (~6 minutes). Additionally, we can jointly train for multiple concepts or combine multiple fine-tuned models into one via closed-form constrained optimization. Our fine-tuned model generates variations of multiple new concepts and seamlessly composes them with existing concepts in novel settings. Our method outperforms or performs on par with several baselines and concurrent works in both qualitative and quantitative evaluations while being memory and computationally efficient.