qa-FLoRA: Data-free query-adaptive Fusion of LoRAs for LLMs

TL;DR

qa-FLoRA introduces a data-free, query-adaptive method for fusing LoRA adapters in large language models, improving multi-domain task performance without additional training or data.

Contribution

It presents a novel, dynamic fusion technique that measures distributional divergence to adaptively combine adapters, eliminating the need for supervised training or composite data.

Findings

Outperforms static fusion by ~5-6% on multilingual tasks.

Surpasses data-intensive baselines by ~7-10%.

Reveals interpretable fusion patterns across layers.

Abstract

The deployment of large language models for specialized tasks often requires domain-specific parameter-efficient finetuning through Low-Rank Adaptation (LoRA) modules. However, effectively fusing these adapters to handle complex, multi-domain composite queries remains a critical challenge. Existing LoRA fusion approaches either use static weights, which assign equal relevance to each participating LoRA, or require data-intensive supervised training for every possible LoRA combination to obtain respective optimal fusion weights. We propose qa-FLoRA, a novel query-adaptive data-and-training-free method for LoRA fusion that dynamically computes layer-level fusion weights by measuring distributional divergence between the base model and respective adapters. Our approach eliminates the need for composite training data or domain-representative samples, making it readily applicable to existing adapter collections. Extensive experiments across nine multilingual composite tasks spanning mathematics, coding, and medical domains, show that qa-FLoRA outperforms static fusion by ~5% with LLaMA-2 and ~6% with LLaMA-3, and the training-free baselines by ~7% with LLaMA-2 and ~10% with LLaMA-3, while significantly closing the gap with supervised baselines. Further, layer-level analysis of our fusion weights reveals interpretable fusion patterns, demonstrating the effectiveness of our approach for robust multi-domain adaptation.