Refining Salience-Aware Sparse Fine-Tuning Strategies for Language Models

TL;DR

This paper systematically evaluates salience metrics for sparsity-based PEFT in language models, finding simple gradient-based static masking strategies effective and efficient, challenging the need for complex methods.

Contribution

It introduces a reliable, computationally efficient salience metric for SPEFT and demonstrates static masking's effectiveness over dynamic masking in NLP tasks.

Findings

Gradient-based salience metrics are reliable and efficient.

Static masking matches dynamic masking in performance.

Simple SPEFT outperforms other fine-tuning methods.

Abstract

Parameter-Efficient Fine-Tuning (PEFT) has gained prominence through low-rank adaptation methods like LoRA. In this paper, we focus on sparsity-based PEFT (SPEFT), which introduces trainable sparse adaptations to the weight matrices in the model, offering greater flexibility in selecting fine-tuned parameters compared to low-rank methods. We conduct the first systematic evaluation of salience metrics for SPEFT, inspired by zero-cost NAS proxies, and identify simple gradient-based metrics is reliable, and results are on par with the best alternatives, offering both computational efficiency and robust performance. Additionally, we compare static and dynamic masking strategies, finding that static masking, which predetermines non-zero entries before training, delivers efficiency without sacrificing performance, while dynamic masking offers no substantial benefits. Across NLP tasks, a simple gradient-based, static SPEFT consistently outperforms other fine-tuning methods for LLMs, providing a simple yet effective baseline for SPEFT. Our work challenges the notion that complexity is necessary for effective PEFT, while our open-source framework establishes a reproducible benchmark for future research, which is available at [https://github.com/0-ml/speft].