LoRA-Gen: Specializing Large Language Model via Online LoRA Generation

TL;DR

LoRA-Gen introduces an online framework that leverages cloud-side models to generate and merge LoRA parameters for edge models, enhancing domain-specific NLP task performance and efficiency without additional training.

Contribution

The paper presents a novel online method for generating LoRA parameters using a large cloud model, enabling efficient specialization of small edge models without retraining.

Findings

Outperforms conventional LoRA fine-tuning in accuracy.

Achieves 2.1x inference speedup on TinyLLaMA-1.1B.

Provides a 10.1x compression ratio on Gemma-2B.

Abstract

Recent advances have highlighted the benefits of scaling language models to enhance performance across a wide range of NLP tasks. However, these approaches still face limitations in effectiveness and efficiency when applied to domain-specific tasks, particularly for small edge-side models. We propose the LoRA-Gen framework, which utilizes a large cloud-side model to generate LoRA parameters for edge-side models based on task descriptions. By employing the reparameterization technique, we merge the LoRA parameters into the edge-side model to achieve flexible specialization. Our method facilitates knowledge transfer between models while significantly improving the inference efficiency of the specialized model by reducing the input context length. Without specialized training, LoRA-Gen outperforms conventional LoRA fine-tuning, which achieves competitive accuracy and a 2.1x speedup with TinyLLaMA-1.1B in reasoning tasks. Besides, our method delivers a compression ratio of 10.1x with Gemma-2B on intelligent agent tasks.