AdaSVD: Adaptive Singular Value Decomposition for Large Language Models

TL;DR

AdaSVD is an adaptive SVD-based compression method for large language models that intelligently compensates for truncation errors and assigns layer-specific compression ratios, leading to better performance and memory efficiency.

Contribution

AdaSVD introduces adaptive error compensation and layer-wise compression ratios, improving upon existing SVD-based LLM compression techniques.

Findings

Outperforms state-of-the-art SVD-based methods across multiple models.

Achieves significant memory reduction with minimal performance loss.

Demonstrates effectiveness across various LLM and VLM benchmarks.

Abstract

Large language models (LLMs) have achieved remarkable success in natural language processing (NLP) tasks, yet their substantial memory requirements present significant challenges for deployment on resource-constrained devices. Singular Value Decomposition (SVD) has emerged as a promising compression technique for LLMs, offering considerable reductions in memory overhead. However, existing SVD-based methods often struggle to effectively mitigate the errors introduced by SVD truncation, leading to a noticeable performance gap when compared to the original models. Furthermore, applying a uniform compression ratio across all transformer layers fails to account for the varying importance of different layers. To address these challenges, we propose AdaSVD, an adaptive SVD-based LLM compression approach. Specifically, AdaSVD introduces adaComp, which adaptively compensates for SVD truncation errors by alternately updating the singular matrices $\mathcal{U}$ and $\mathcal{V}^\top$. Additionally, AdaSVD introduces adaCR, which adaptively assigns layer-specific compression ratios based on the relative importance of each layer. Extensive experiments across multiple LLM/VLM families and evaluation metrics demonstrate that AdaSVD consistently outperforms state-of-the-art (SOTA) SVD-based methods, achieving superior performance with significantly reduced memory requirements. Code and models of AdaSVD will be available at https://github.com/ZHITENGLI/AdaSVD.