Task-Aware LoRA Adapter Composition via Similarity Retrieval in Vector Databases

TL;DR

This paper introduces a retrieval-based framework for dynamically composing LoRA adapters for unseen NLP tasks, enabling zero-shot generalization and efficient multitask learning without retraining models.

Contribution

The paper proposes a novel method that uses similarity retrieval in vector databases to compose and merge LoRA adapters dynamically for diverse NLP tasks.

Findings

Retrieval-based adapter merging often matches or exceeds task-specific fine-tuning performance.

Linear merging achieves 70.95% on PIQA and 77.62% on RTE, outperforming baselines.

The approach requires no retraining of the retriever and uses frozen embeddings.

Abstract

Parameter efficient fine tuning methods like LoRA have enabled task specific adaptation of large language models, but efficiently composing multiple specialized adapters for unseen tasks remains challenging. We present a novel framework for dynamic LoRA adapter composition that leverages similarity retrieval in vector databases to enable zero-shot generalization across diverse NLP tasks. Our approach constructs a task-aware vector database by embedding training examples from 22 datasets spanning commonsense reasoning, question answering, natural language inference, and sentiment analysis. At inference time, we retrieve the most similar training examples, compute task similarity distributions via nucleus sampling, and dynamically merge relevant LoRA adapters using retrieval weighted fusion strategies. We evaluated four merging methods Linear, Concatenation, TIES, and Magnitude Prune demonstrating that our dataset centric retrieval approach often matches or exceeds the performance of individually fine-tuned task-specific adapters. Notably, Linear merging achieves 70.95% on PIQA and 77.62% on RTE, substantially outperforming single-task baselines (46% and 52%, respectively). Our framework requires no additional retriever training, operates with frozen embeddings, and enables efficient, interpretable adapter composition. These results suggest that retrieval based dynamic merging offers a promising direction for scalable, parameter-efficient multitask learning without requiring full model retraining for each new task.