Toffee: Efficient Million-Scale Dataset Construction for Subject-Driven Text-to-Image Generation

TL;DR

This paper introduces Toffee, a cost-effective method for constructing large-scale subject-driven text-to-image datasets without subject-level fine-tuning, enabling high-quality image generation and editing with significantly reduced GPU hours.

Contribution

We propose Toffee, a novel dataset construction approach that generates millions of high-quality image pairs without fine-tuning, vastly reducing costs and enabling large-scale subject-driven image tasks.

Findings

Constructed a 5 million image pair dataset, five times larger than previous datasets.

Achieved comparable results in subject-driven image editing and generation using our dataset.

Reduced dataset construction cost by tens of thousands of GPU hours.

Abstract

In subject-driven text-to-image generation, recent works have achieved superior performance by training the model on synthetic datasets containing numerous image pairs. Trained on these datasets, generative models can produce text-aligned images for specific subject from arbitrary testing image in a zero-shot manner. They even outperform methods which require additional fine-tuning on testing images. However, the cost of creating such datasets is prohibitive for most researchers. To generate a single training pair, current methods fine-tune a pre-trained text-to-image model on the subject image to capture fine-grained details, then use the fine-tuned model to create images for the same subject based on creative text prompts. Consequently, constructing a large-scale dataset with millions of subjects can require hundreds of thousands of GPU hours. To tackle this problem, we propose Toffee, an efficient method to construct datasets for subject-driven editing and generation. Specifically, our dataset construction does not need any subject-level fine-tuning. After pre-training two generative models, we are able to generate infinite number of high-quality samples. We construct the first large-scale dataset for subject-driven image editing and generation, which contains 5 million image pairs, text prompts, and masks. Our dataset is 5 times the size of previous largest dataset, yet our cost is tens of thousands of GPU hours lower. To test the proposed dataset, we also propose a model which is capable of both subject-driven image editing and generation. By simply training the model on our proposed dataset, it obtains competitive results, illustrating the effectiveness of the proposed dataset construction framework.