SPP: Sparsity-Preserved Parameter-Efficient Fine-Tuning for Large Language Models

TL;DR

This paper introduces SPP, a novel fine-tuning method that preserves sparsity in large language models, significantly improving performance of highly sparse models while maintaining their structure and efficiency.

Contribution

SPP employs learnable matrices to optimize sparse LLM weights, maintaining sparsity patterns and outperforming existing pruning methods in performance retention.

Findings

Enhances performance of sparse LLMs with high sparsity ratios.

Maintains model sparsity pattern during training and weight merging.

Effective on LLaMA and LLaMA-2 models with various sparsity patterns.

Abstract

Large Language Models (LLMs) have become pivotal in advancing the field of artificial intelligence, yet their immense sizes pose significant challenges for both fine-tuning and deployment. Current post-training pruning methods, while reducing the sizes of LLMs, often fail to maintain their original performance. To address these challenges, this paper introduces SPP, a Sparsity-Preserved Parameter-efficient fine-tuning method. Different from existing post-training pruning approaches that struggle with performance retention, SPP proposes to employ lightweight learnable column and row matrices to optimize sparse LLM weights, keeping the structure and sparsity of pruned pre-trained models intact. By element-wise multiplication and residual addition, SPP ensures the consistency of model sparsity pattern and ratio during both training and weight-merging processes. We demonstrate the effectiveness of SPP by applying it to the LLaMA and LLaMA-2 model families with recent post-training pruning methods. Our results show that SPP significantly enhances the performance of models with different sparsity patterns (i.e. unstructured and N:M sparsity), especially for those with high sparsity ratios (e.g. 75%), making it a promising solution for the efficient fine-tuning of sparse LLMs. Code will be made available at https://github.com/Lucky-Lance/SPP.