KaSA: Knowledge-Aware Singular-Value Adaptation of Large Language Models

TL;DR

KaSA introduces a knowledge-aware singular-value adaptation method for large language models, improving task-specific performance by dynamically activating relevant knowledge during fine-tuning, outperforming existing PEFT methods across multiple benchmarks.

Contribution

We propose KaSA, a novel PEFT approach using knowledge-aware SVD to selectively activate relevant knowledge, enhancing LLM adaptation efficiency and effectiveness.

Findings

KaSA outperforms FFT and 14 PEFT baselines on 16 benchmarks.

KaSA demonstrates consistent improvements across NLU, NLG, and reasoning tasks.

The method effectively activates relevant knowledge, improving model performance.

Abstract

The increasing sizes of large language models (LLMs) result in significant computational overhead and memory usage when adapting these models to specific tasks or domains. Various parameter-efficient fine-tuning (PEFT) methods have been devised to mitigate these challenges by training a small set of parameters for the task-specific updates of the model weights. Among PEFT methods, LoRA stands out for its simplicity and efficiency, inspiring the development of a series of variants. However, LoRA and its successors disregard the knowledge that is noisy or irrelevant to the targeted task, detrimentally impacting model performance and leading to suboptimality. To address this limitation, we introduce Knowledge-aware Singular-value Adaptation (KaSA), a PEFT method that leverages singular value decomposition (SVD) with knowledge-aware singular values to dynamically activate knowledge based on its relevance to the task at hand. We conduct extensive experiments across a range of LLMs on tasks spanning natural language understanding (NLU), generation (NLG), instruction following, and commonsense reasoning. The experimental results demonstrate that KaSA consistently outperforms FFT and 14 popular PEFT baselines across 16 benchmarks and 4 synthetic datasets, underscoring our method's efficacy and adaptability. The source code of our method is available at https://github.com/juyongjiang/KaSA.