ZipLoRA: Any Subject in Any Style by Effectively Merging LoRAs

TL;DR

ZipLoRA is a novel method that effectively merges independently trained style and subject LoRAs, enabling high-fidelity generation of any subject in any style with improved consistency and recontextualization capabilities.

Contribution

It introduces a cheap and effective approach to merge LoRAs for personalized generation, addressing limitations of previous methods in subject and style fidelity.

Findings

Significant improvement in subject and style fidelity over baselines

Ability to generate any subject in any style with meaningful quality

Preserves recontextualization capabilities

Abstract

Methods for finetuning generative models for concept-driven personalization generally achieve strong results for subject-driven or style-driven generation. Recently, low-rank adaptations (LoRA) have been proposed as a parameter-efficient way of achieving concept-driven personalization. While recent work explores the combination of separate LoRAs to achieve joint generation of learned styles and subjects, existing techniques do not reliably address the problem; they often compromise either subject fidelity or style fidelity. We propose ZipLoRA, a method to cheaply and effectively merge independently trained style and subject LoRAs in order to achieve generation of any user-provided subject in any user-provided style. Experiments on a wide range of subject and style combinations show that ZipLoRA can generate compelling results with meaningful improvements over baselines in subject and style fidelity while preserving the ability to recontextualize. Project page: https://ziplora.github.io