A Survey on LoRA of Large Language Models

TL;DR

This survey comprehensively reviews recent advances in Low-Rank Adaptation (LoRA) for large language models, covering improvements, generalization, efficiency, privacy, and applications, highlighting future research directions.

Contribution

It provides a structured overview of LoRA research, categorizing progress and identifying future directions in this rapidly growing field.

Findings

LoRA enhances parameter-efficient fine-tuning of large models.

Cross-task generalization with LoRA improves multi-task performance.

Efficiency and privacy methods expand LoRA's practical applications.

Abstract

Low-Rank Adaptation~(LoRA), which updates the dense neural network layers with pluggable low-rank matrices, is one of the best performed parameter efficient fine-tuning paradigms. Furthermore, it has significant advantages in cross-task generalization and privacy-preserving. Hence, LoRA has gained much attention recently, and the number of related literature demonstrates exponential growth. It is necessary to conduct a comprehensive overview of the current progress on LoRA. This survey categorizes and reviews the progress from the perspectives of (1) downstream adaptation improving variants that improve LoRA's performance on downstream tasks; (2) cross-task generalization methods that mix multiple LoRA plugins to achieve cross-task generalization; (3) efficiency-improving methods that boost the computation-efficiency of LoRA; (4) data privacy-preserving methods that use LoRA in federated learning; (5) application. Besides, this survey also discusses the future directions in this field. At last, we provide a Github page~\footnote{\href{https://github.com/ZJU-LLMs/Awesome-LoRAs.git}{https://github.com/ZJU-LLMs/Awesome-LoRAs.git}} for readers to check the updates and initiate discussions on this survey paper.