SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression

TL;DR

SVD-LLM introduces a truncation-aware SVD-based compression method for large language models that maintains accuracy at high compression ratios by incorporating data whitening and parameter updates.

Contribution

It proposes a novel SVD-based post-training compression technique with truncation-aware data whitening and sequential low-rank approximation for improved LLM compression.

Findings

Outperforms existing methods at high compression ratios

Effective across multiple datasets and LLM models

Reduces accuracy loss through parameter updates

Abstract

The advancements in Large Language Models (LLMs) have been hindered by their substantial sizes, which necessitates LLM compression methods for practical deployment. Singular Value Decomposition (SVD) offers a promising solution for LLM compression. However, state-of-the-art SVD-based LLM compression methods have two key limitations: truncating smaller singular values may lead to higher compression loss, and the lack of update on the compressed weights after SVD truncation. In this work, we propose SVD-LLM, a SVD-based post-training LLM compression method that addresses the limitations of existing methods. SVD-LLM incorporates a truncation-aware data whitening technique to ensure a direct mapping between singular values and compression loss. Moreover, SVD-LLM adopts a parameter update with sequential low-rank approximation to compensate for the accuracy degradation after SVD compression. We evaluate SVD-LLM on 10 datasets and seven models from three different LLM families at three different scales. Our results demonstrate the superiority of SVD-LLM over state-of-the-arts, especially at high model compression ratios. Our code is available at https://github.com/AIoT-MLSys-Lab/SVD-LLM