Not All Adapters Matter: Selective Adapter Freezing for Memory-Efficient Fine-Tuning of Language Models

TL;DR

This paper introduces SAFE, a method for selectively freezing adapters in large language models during fine-tuning, significantly reducing resource consumption while maintaining or improving task performance.

Contribution

SAFE is a novel selective freezing technique that identifies and freezes less important adapters early, optimizing resource use without sacrificing accuracy.

Findings

SAFE reduces memory, computation, and training time by over 40%.

SAFE achieves comparable or better performance than baseline methods.

Selective freezing induces regularization, improving generalization.

Abstract

Transformer-based large-scale pre-trained models achieve great success. Fine-tuning is the standard practice for leveraging these models in downstream tasks. Among the fine-tuning methods, adapter-tuning provides a parameter-efficient fine-tuning by introducing lightweight trainable modules while keeping most pre-trained parameters frozen. However, existing adapter-tuning methods still impose substantial resource usage. Through our investigation, we show that each adapter unequally contributes to both task performance and resource usage. Motivated by this insight, we propose Selective Adapter FrEezing (SAFE), which gradually freezes less important adapters early to reduce unnecessary resource usage while maintaining performance. In our experiments, SAFE reduces memory usage, computation amount, and training time by 42.85\%, 34.59\%, and 11.82\%, respectively, while achieving comparable or better task performance compared to the baseline. We also demonstrate that SAFE induces regularization effect, thereby smoothing the loss landscape, which enables the model to generalize better by avoiding sharp minima.