GIFT-SW: Gaussian noise Injected Fine-Tuning of Salient Weights for LLMs

TL;DR

GIFT-SW is a novel parameter-efficient fine-tuning method for LLMs that selectively updates salient weights and injects noise into others, improving performance and robustness with less computation.

Contribution

The paper introduces GIFT-SW, a new PEFT approach that identifies and fine-tunes salient weights while injecting noise into non-salient ones, outperforming existing methods.

Findings

GIFT-SW surpasses full fine-tuning and modern PEFT methods in experiments.

It effectively recovers performance in quantized models.

The method is computationally efficient and practical.

Abstract

Parameter Efficient Fine-Tuning (PEFT) methods have gained popularity and democratized the usage of Large Language Models (LLMs). Recent studies have shown that a small subset of weights significantly impacts performance. Based on this observation, we introduce a novel PEFT method, called Gaussian noise Injected Fine Tuning of Salient Weights (GIFT-SW). Our method updates only salient columns, while injecting Gaussian noise into non-salient ones. To identify these columns, we developeda generalized sensitivity metric that extends and unifies metrics from previous studies. Experiments with LLaMA models demonstrate that GIFT-SW outperforms full fine-tuning and modern PEFT methods under the same computational budget. Moreover, GIFT-SW offers practical advantages to recover performance of models subjected to mixed-precision quantization with keeping salient weights in full precision.