LoRA on the Go: Instance-level Dynamic LoRA Selection and Merging

TL;DR

LoGo is a training-free, dynamic framework that selects and merges LoRA adapters at the instance level, enhancing large language model adaptability across diverse NLP tasks without additional training.

Contribution

Introduces LoGo, a novel method for real-time adapter selection and merging without training, improving efficiency and performance in multi-domain NLP applications.

Findings

LoGo outperforms training-based baselines on some tasks by up to 3.6%.

LoGo maintains inference throughput while adapting to diverse datasets.

LoGo is effective across multiple NLP benchmarks, datasets, and model families.

Abstract

Low-Rank Adaptation (LoRA) has emerged as a parameter-efficient approach for fine-tuning large language models. However, conventional LoRA adapters are typically trained for a single task, limiting their applicability in real-world settings where inputs may span diverse and unpredictable domains. At inference time, existing approaches combine multiple LoRAs for improving performance on diverse tasks, while usually requiring labeled data or additional task-specific training, which is expensive at scale. In this work, we introduce LoRA on the Go (LoGo), a training-free framework that dynamically selects and merges adapters at the instance level without any additional requirements. LoGo leverages signals extracted from a single forward pass through LoRA adapters, to identify the most relevant adapters and determine their contributions on-the-fly. Across 5 NLP benchmarks, 27 datasets, and 3 model families, LoGo outperforms training-based baselines on some tasks upto a margin of 3.6% while remaining competitive on other tasks and maintaining inference throughput, highlighting its effectiveness and practicality.