LoSiA: Efficient High-Rank Fine-Tuning via Subnet Localization and Optimization

TL;DR

LoSiA introduces a dynamic subnet localization method for efficient high-rank fine-tuning, significantly reducing training time and computational costs while maintaining performance.

Contribution

It proposes a novel subnet localization and optimization approach for PEFT, improving efficiency and effectiveness in high-rank fine-tuning tasks.

Findings

LoSiA reduces training latency by 27% compared to LoRA.

Achieves minimal performance drop relative to full fine-tuning.

Reduces forgetting during continued training.

Abstract

Parameter-Efficient Fine-Tuning (PEFT) methods, such as LoRA, significantly reduce the number of trainable parameters by introducing low-rank decomposition matrices. However, existing methods perform extensive matrix multiplications in domain specialization tasks, resulting in computational inefficiency and sub-optimal fine-tuning performance. Hence, we propose LoSiA(Low-Resources Subnet Integration Adaptation), an innovative method that dynamically localizes and optimizes critical parameters during the training process. Specifically, it identifies a sub-network using gradient sparsity analysis and optimizes it as the trainable target. This design enables effective high-rank adaptation by updating only the sub-network parameters, reducing the additional matrix multiplication. We also present LoSiA-Pro, a faster implementation of LoSiA, which reduces the training latency by about $27\%$ compared to LoRA. Extensive evaluations show that our method achieves minimal performance drop compared to full fine-tuning, while requiring the least training time across domain specialization and common-sense reasoning tasks. Further analysis shows that LoSiA also reduces forgetting during continued training. The source code is available at https://github.com/KlozeWang/LoSiA.