LoRA.rar: Learning to Merge LoRAs via Hypernetworks for Subject-Style Conditioned Image Generation

TL;DR

LoRA.rar introduces a hypernetwork-based approach for rapid, high-quality merging of style and subject LoRAs in personalized image generation, achieving over 4000x speedup and better evaluation metrics.

Contribution

We propose a hypernetwork trained on style and subject LoRAs to enable fast, high-quality personalization in image generation, surpassing existing methods in speed and fidelity.

Findings

Over 4000x faster LoRA merging process.

Outperforms state-of-the-art in content and style quality.

MLLM-based evaluation correlates well with human judgments.

Abstract

Recent advancements in image generation models have enabled personalized image creation with both user-defined subjects (content) and styles. Prior works achieved personalization by merging corresponding low-rank adapters (LoRAs) through optimization-based methods, which are computationally demanding and unsuitable for real-time use on resource-constrained devices like smartphones. To address this, we introduce LoRA$.$rar, a method that not only improves image quality but also achieves a remarkable speedup of over $4000\times$ in the merging process. We collect a dataset of style and subject LoRAs and pre-train a hypernetwork on a diverse set of content-style LoRA pairs, learning an efficient merging strategy that generalizes to new, unseen content-style pairs, enabling fast, high-quality personalization. Moreover, we identify limitations in existing evaluation metrics for content-style quality and propose a new protocol using multimodal large language models (MLLMs) for more accurate assessment. Our method significantly outperforms the current state of the art in both content and style fidelity, as validated by MLLM assessments and human evaluations.