T-LoRA: Single Image Diffusion Model Customization Without Overfitting

TL;DR

T-LoRA introduces a timestep-sensitive low-rank adaptation method for single-image diffusion model customization, effectively preventing overfitting and enhancing output diversity and fidelity.

Contribution

The paper presents T-LoRA, a novel framework that dynamically adjusts fine-tuning based on diffusion timesteps and ensures component independence, improving single-image diffusion model personalization.

Findings

Outperforms standard LoRA and other methods in experiments.

Achieves better balance between concept fidelity and text alignment.

Effective in preventing overfitting at higher diffusion timesteps.

Abstract

While diffusion model fine-tuning offers a powerful approach for customizing pre-trained models to generate specific objects, it frequently suffers from overfitting when training samples are limited, compromising both generalization capability and output diversity. This paper tackles the challenging yet most impactful task of adapting a diffusion model using just a single concept image, as single-image customization holds the greatest practical potential. We introduce T-LoRA, a Timestep-Dependent Low-Rank Adaptation framework specifically designed for diffusion model personalization. We show that higher diffusion timesteps are more prone to overfitting than lower ones, necessitating a timestep-sensitive fine-tuning strategy. T-LoRA incorporates two key innovations: (1) a dynamic fine-tuning strategy that adjusts rank-constrained updates based on diffusion timesteps, and (2) a weight parametrization technique that ensures independence between adapter components through orthogonal initialization. Extensive experiments on SD-XL and FLUX-1.dev show that T-LoRA and its individual components outperform standard LoRA and other diffusion model personalization techniques, achieving a superior balance between concept fidelity and text alignment. Project page is available at https://controlgenai.github.io/T-LoRA/.