Position: Pause Recycling LoRAs and Prioritize Mechanisms to Uncover Limits and Effectiveness

TL;DR

This paper critically examines the effectiveness of reusing low-rank adapters (LoRAs) in large language models, highlighting their limitations and advocating for a focus on understanding when such reuse is genuinely beneficial.

Contribution

It shifts the research focus from developing new LoRA merging algorithms to analyzing the conditions under which reuse is effective, supported by theoretical and empirical evidence.

Findings

Reusing LoRAs often fails to achieve true compositional generalization.

Parameter averaging and dynamic selection do not reliably integrate knowledge across datasets.

LoRA's limited expressiveness constrains its reuse effectiveness.

Abstract

Merging or routing low-rank adapters (LoRAs) has emerged as a popular solution for enhancing large language models, particularly when data access is restricted by regulatory or domain-specific constraints. This position paper argues that the research community should shift its focus from developing new merging or routing algorithms to understanding the conditions under which reusing LoRAs is truly effective. Through theoretical analysis and synthetic two-hop reasoning and math word-problem tasks, we examine whether reusing LoRAs enables genuine compositional generalization or merely reflects shallow pattern matching. Evaluating two data-agnostic methods--parameter averaging and dynamic adapter selection--we found that reusing LoRAs often fails to logically integrate knowledge across disjoint fine-tuning datasets, especially when such knowledge is underrepresented during pretraining. Our empirical results, supported by theoretical insights into LoRA's limited expressiveness, highlight the preconditions and constraints of reusing them for unseen tasks and cast doubt on its feasibility as a truly data-free approach. We advocate for pausing the pursuit of novel methods for recycling LoRAs and emphasize the need for rigorous mechanisms to guide future academic research in adapter-based model merging and practical system designs for practitioners.