Zero-Shot Adaptation of Parameter-Efficient Fine-Tuning in Diffusion Models

TL;DR

ProLoRA enables zero-shot transfer of low-rank fine-tuning adjustments in diffusion models, allowing adaptation to new models without retraining or additional data, thus improving flexibility and efficiency.

Contribution

ProLoRA introduces a novel method for zero-shot adaptation of parameter-efficient fine-tuning in diffusion models by projecting source adjustments into target models' weight space.

Findings

Successful knowledge transfer demonstrated on text-to-image models

Achieves comparable performance without retraining

Overcomes data constraints of traditional methods

Abstract

We introduce ProLoRA, enabling zero-shot adaptation of parameter-efficient fine-tuning in text-to-image diffusion models. ProLoRA transfers pre-trained low-rank adjustments (e.g., LoRA) from a source to a target model without additional training data. This overcomes the limitations of traditional methods that require retraining when switching base models, often challenging due to data constraints. ProLoRA achieves this via projection of source adjustments into the target model's weight space, leveraging subspace and null space similarities and selectively targeting aligned layers. Evaluations on established text-to-image models demonstrate successful knowledge transfer and comparable performance without retraining.