LoFT: LoRA-fused Training Dataset Generation with Few-shot Guidance

TL;DR

LoFT is a novel dataset generation framework that fine-tunes LoRA weights on real images to produce synthetic data with enhanced fidelity and diversity, improving supervised learning performance.

Contribution

Introducing LoFT, a new method that fuses LoRA weights on real images for high-quality synthetic dataset generation with few-shot guidance.

Findings

LoFT-generated data outperforms other synthetic datasets in accuracy.

Training with LoFT data scales well with dataset size.

LoFT produces datasets with high fidelity and diversity.

Abstract

Despite recent advances in text-to-image generation, using synthetically generated data seldom brings a significant boost in performance for supervised learning. Oftentimes, synthetic datasets do not faithfully recreate the data distribution of real data, i.e., they lack the fidelity or diversity needed for effective downstream model training. While previous work has employed few-shot guidance to address this issue, existing methods still fail to capture and generate features unique to specific real images. In this paper, we introduce a novel dataset generation framework named LoFT, LoRA-Fused Training-data Generation with Few-shot Guidance. Our method fine-tunes LoRA weights on individual real images and fuses them at inference time, producing synthetic images that combine the features of real images for improved diversity and fidelity of generated data. We evaluate the synthetic data produced by LoFT on 10 datasets, using 8 to 64 real images per class as guidance and scaling up to 1000 images per class. Our experiments show that training on LoFT-generated data consistently outperforms other synthetic dataset methods, significantly increasing accuracy as the dataset size increases. Additionally, our analysis demonstrates that LoFT generates datasets with high fidelity and sufficient diversity, which contribute to the performance improvement. The code is available at https://github.com/ExplainableML/LoFT.