Least but not Last: Fine-tuning Intermediate Principal Components for Better Performance-Forgetting Trade-Offs

TL;DR

This paper analyzes the trade-offs in low-rank adaptation of large models, showing that fine-tuning intermediate principal components improves performance and knowledge retention across tasks.

Contribution

It introduces a novel initialization method focusing on intermediate components, enhancing the balance between task performance and forgetting in LoRA.

Findings

Fine-tuning intermediate components yields better performance-forgetting balance.

The proposed approach is more robust to high learning rates.

Empirical results show improved accuracy and reduced forgetting in vision and NLP tasks.

Abstract

Low-Rank Adaptation (LoRA) methods have emerged as crucial techniques for adapting large pre-trained models to downstream tasks under computational and memory constraints. However, they face a fundamental challenge in balancing task-specific performance gains against catastrophic forgetting of pre-trained knowledge, where existing methods provide inconsistent recommendations. This paper presents a comprehensive analysis of the performance-forgetting trade-offs inherent in low-rank adaptation using principal components as initialization. Our investigation reveals that fine-tuning intermediate components leads to better balance and show more robustness to high learning rates than first (PiSSA) and last (MiLoRA) components in existing work. Building on these findings, we provide a practical approach for initialization of LoRA that offers superior trade-offs. We demonstrate in a thorough empirical study on a variety of computer vision and NLP tasks that our approach improves accuracy and reduces forgetting, also in continual learning scenarios.