Retrieval-Augmented Mixture of LoRA Experts for Uploadable Machine Learning

TL;DR

This paper introduces RAMoLE, a retrieval-augmented framework that adaptively composes multiple LoRA adapters for personalized, mixed-task LLM services in Uploadable Machine Learning platforms, improving performance and scalability.

Contribution

It proposes a novel retrieval-based method for dynamically selecting and combining LoRA adapters, addressing the challenges of unseen adapters and mixed tasks in UML environments.

Findings

RAMoLE outperforms baseline methods in experiments.

The framework effectively handles heterogeneous and unseen LoRAs.

Scalable and adaptable to evolving UML platforms.

Abstract

Low-Rank Adaptation (LoRA) offers an efficient way to fine-tune large language models (LLMs). Its modular and plug-and-play nature allows the integration of various domain-specific LoRAs, enhancing LLM capabilities. Open-source platforms like Huggingface and Modelscope have introduced a new computational paradigm, Uploadable Machine Learning (UML). In UML, contributors use decentralized data to train specialized adapters, which are then uploaded to a central platform to improve LLMs. This platform uses these domain-specific adapters to handle mixed-task requests requiring personalized service. Previous research on LoRA composition either focuses on specific tasks or fixes the LoRA selection during training. However, in UML, the pool of LoRAs is dynamically updated with new uploads, requiring a generalizable selection mechanism for unseen LoRAs. Additionally, the mixed-task nature of downstream requests necessitates personalized services. To address these challenges, we propose Retrieval-Augmented Mixture of LoRA Experts (RAMoLE), a framework that adaptively retrieves and composes multiple LoRAs based on input prompts. RAMoLE has three main components: LoraRetriever for identifying and retrieving relevant LoRAs, an on-the-fly MoLE mechanism for coordinating the retrieved LoRAs, and efficient batch inference for handling heterogeneous requests. Experimental results show that RAMoLE consistently outperforms baselines, highlighting its effectiveness and scalability.