The Appeal and Reality of Recycling LoRAs with Adaptive Merging

TL;DR

This paper investigates the effectiveness of adaptively merging recycled LoRA modules from model repositories, revealing limited benefits over training new LoRAs and highlighting potential regularization effects rather than transfer learning.

Contribution

It provides an empirical analysis of adaptive LoRA merging with recycled modules, introduces a new merging method, and challenges assumptions about transfer learning benefits.

Findings

Adaptive merging improves base model performance but not over training new LoRAs.

Randomly initialized LoRAs perform similarly to trained ones in merging.

Positive transfer occurs only with highly relevant LoRAs in the pool.

Abstract

The widespread availability of fine-tuned LoRA modules for open pre-trained models has led to an interest in methods that can adaptively merge LoRAs to improve performance. These methods typically include some way of selecting LoRAs from a pool and tune merging coefficients based on a task-specific dataset. While adaptive merging methods have demonstrated improvements in some settings, no past work has attempted to recycle LoRAs found "in the wild" on model repositories like the Hugging Face Hub. To address this gap, we consider recycling from a pool of nearly 1,000 user-contributed LoRAs trained from the Llama 3.1 8B-Instruct language model. Our empirical study includes a range of adaptive and non-adaptive merging methods in addition to a new method designed via a wide search over the methodological design space. We demonstrate that adaptive merging methods can improve performance over the base model but provide limited benefit over training a new LoRA on the same data used to set merging coefficients. We additionally find not only that the specific choice of LoRAs to merge has little importance, but that using LoRAs with randomly initialized parameter values yields similar performance. This raises the possibility that adaptive merging from recycled LoRAs primarily works via some kind of regularization effect, rather than by enabling positive cross-task transfer. To better understand why past work has proven successful, we confirm that positive transfer is indeed possible when there are highly relevant LoRAs in the pool. We release the model checkpoints and code online.