DiffLoRA: Generating Personalized Low-Rank Adaptation Weights with Diffusion

TL;DR

DiffLoRA is an efficient diffusion-based method that predicts personalized low-rank adaptation weights for zero-shot text-to-image personalization, outperforming existing methods in quality and speed without additional fine-tuning.

Contribution

It introduces DiffLoRA, a diffusion model-based hypernetwork that predicts personalized LoRA weights, enabling fast, high-fidelity zero-shot personalization without post-processing.

Findings

Outperforms existing personalization methods on multiple benchmarks.

Achieves high identity fidelity with improved efficiency.

Eliminates need for post-processing optimization.

Abstract

Personalized text-to-image generation has gained significant attention for its capability to generate high-fidelity portraits of specific identities conditioned on user-defined prompts. Existing methods typically involve test-time fine-tuning or incorporating an additional pre-trained branch. However, these approaches struggle to simultaneously address efficiency, identity fidelity, and the preservation of the model's original generative capabilities. In this paper, we propose DiffLoRA, an efficient method that leverages the diffusion model as a hypernetwork to predict personalized Low-Rank Adaptation (LoRA) weights based on the reference images. By incorporating these LoRA weights into the off-the-shelf text-to-image model, DiffLoRA enables zero-shot personalization during inference, eliminating the need for post-processing optimization. Moreover, we introduce a novel identity-oriented LoRA weights construction pipeline to facilitate the training process of DiffLoRA. The dataset generated through this pipeline enables DiffLoRA to produce consistently high-quality LoRA weights. Notably, the distinctive properties of the diffusion model enhance the generation of superior weights by employing probabilistic modeling to capture intricate structural patterns and thoroughly explore the weight space. Comprehensive experimental results demonstrate that DiffLoRA outperforms existing personalization approaches across multiple benchmarks, achieving both time efficiency and maintaining identity fidelity throughout the personalization process.