Subject-driven Text-to-Image Generation via Apprenticeship Learning

TL;DR

SuTI introduces a fast, subject-driven text-to-image generation method that uses apprenticeship learning to imitate expert models, enabling instant, high-quality, and customizable images without subject-specific fine-tuning.

Contribution

The paper presents SuTI, a novel in-context learning approach that replaces expensive fine-tuning with apprenticeship learning, significantly speeding up subject-specific image generation.

Findings

SuTI generates images 20x faster than state-of-the-art optimization methods.

Human evaluation shows SuTI outperforms existing models on subject and text alignment.

SuTI effectively imitates expert models trained on millions of subject-specific image clusters.

Abstract

Recent text-to-image generation models like DreamBooth have made remarkable progress in generating highly customized images of a target subject, by fine-tuning an ``expert model'' for a given subject from a few examples. However, this process is expensive, since a new expert model must be learned for each subject. In this paper, we present SuTI, a Subject-driven Text-to-Image generator that replaces subject-specific fine tuning with in-context learning. Given a few demonstrations of a new subject, SuTI can instantly generate novel renditions of the subject in different scenes, without any subject-specific optimization. SuTI is powered by apprenticeship learning, where a single apprentice model is learned from data generated by a massive number of subject-specific expert models. Specifically, we mine millions of image clusters from the Internet, each centered around a specific visual subject. We adopt these clusters to train a massive number of expert models, each specializing in a different subject. The apprentice model SuTI then learns to imitate the behavior of these fine-tuned experts. SuTI can generate high-quality and customized subject-specific images 20x faster than optimization-based SoTA methods. On the challenging DreamBench and DreamBench-v2, our human evaluation shows that SuTI significantly outperforms existing models like InstructPix2Pix, Textual Inversion, Imagic, Prompt2Prompt, Re-Imagen and DreamBooth, especially on the subject and text alignment aspects.