Subject or Style: Adaptive and Training-Free Mixture of LoRAs

TL;DR

EST-LoRA introduces a training-free, adaptive fusion method for Low-Rank Adaptation models that dynamically balances subject and style influences, improving generation quality and speed.

Contribution

It proposes a novel, training-free adaptive fusion technique considering energy, style discrepancy, and time steps, outperforming existing methods in style and subject balance.

Findings

Outperforms state-of-the-art methods in qualitative and quantitative evaluations.

Achieves faster generation speeds than other fusion approaches.

Effectively balances subject and style contributions during generation.

Abstract

Fine-tuning models via Low-Rank Adaptation (LoRA) demonstrates remarkable performance in subject-driven or style-driven generation tasks. Studies have explored combinations of different LoRAs to jointly generate learned styles and content. However, current methods struggle to balance the original subject and style, and often require additional training. Recently, K-LoRA proposed a training-free LoRA fusion method. But it involves multiple hyperparameters, making it difficult to adapt to all styles and subjects. In this paper, we propose EST-LoRA, a training-free adaptive LoRA fusion method. It comprehensively considers three critical factors: \underline{E}nergy of matrix, \underline{S}tyle discrepancy scores and \underline{T}ime steps. Analogous to the Mixture of Experts (MoE) architecture, the model adaptively selects between subject LoRA and style LoRA within each attention layer. This integrated selection mechanism ensures balanced contributions from both components during the generation process. Experimental results show that EST-LoRA outperforms state-of-the-art methods in both qualitative and quantitative evaluations and achieves faster generation speed compared to other efficient fusion approaches. Our code is publicly available at: https://anonymous.4open.science/r/EST-LoRA-F318.