Effective LoRA Adapter Routing using Task Representations

TL;DR

LORAUTER is a new routing framework for large language models that uses task representations to efficiently select and compose LoRA adapters, outperforming baselines and scaling well to large adapter pools.

Contribution

LORAUTER introduces task-based routing for LoRA adapters, eliminating the need for adapter training data and improving scalability and performance across diverse tasks.

Findings

LORAUTER outperforms baseline routing methods.

Achieves state-of-the-art results on unseen tasks.

Scales effectively to over 1500 adapters.

Abstract

Low-rank adaptation (LoRA) enables parameter efficient specialization of large language models (LLMs) through modular adapters, resulting in rapidly growing public adapter pools spanning diverse tasks. Effectively using these adapters requires routing: selecting and composing the appropriate adapters for a query. We introduce LORAUTER, a novel routing framework that selects and composes LoRA adapters using task representations rather than adapter characteristics. Unlike existing approaches that map queries directly to adapters, LORAUTER routes queries via task embeddings derived from small validation sets and does not require adapter training data. By operating at the task level, LORAUTER achieves efficient routing that scales with the number of tasks rather than the number of adapters. Experiments across multiple tasks show that LORAUTER consistently outperforms baseline routing approaches, matching Oracle performance (101.2%) when task-aligned adapters exist and achieving state-of-the-art results on unseen tasks (+5.2 points). We further demonstrate the robustness of LORAUTER to very large, noisy adapter pools by scaling it to over 1500 adapters.