Beyond Uniform SVD:Dual-Level Optimization across Columns and Modules for LLM Compression

TL;DR

Duo-SVD introduces a dual-level, training-free optimization framework for LLM compression that selectively applies low-rank approximation based on component importance, outperforming existing methods.

Contribution

The paper proposes Duo-SVD, a novel dual-level optimization approach that considers component-specific errors and importance, improving LLM compression beyond traditional SVD techniques.

Findings

Duo-SVD outperforms state-of-the-art SVD-based and pruning methods.

The framework effectively retains critical components while compressing less important ones.

Experimental results show significant improvements in model efficiency and accuracy.

Abstract

Low-rank decomposition, particularly Singular Value Decomposition (SVD), is a pivotal technique for mitigating the storage and computational demands of Large Language Models (LLMs). However, prevalent SVD-based approaches overlook the critical phenomenon that decomposition errors exhibit significant disparity across different components of the parameter matrix, often leading to suboptimal approximation. Furthermore, existing methods lack a direct metric to evaluate the importance of individual weight matrices. To address these limitations, we propose Duo-SVD (Dual-level Optimization SVD), a novel training-free framework that synergizes optimization at both the column and the module levels. First, Duo-SVD incorporates a Column-Preserving Strategy that explicitly retains columns exhibiting high decomposition errors, while applying low-rank approximation solely to those with lower errors. Second, at the module level, we employ a Module-Adaptive Allocation Strategy that formulates ratio allocation as a global constrained optimization problem based on perturbation-induced model deviation. Extensive experiments demonstrate that Duo-SVD consistently outperforms state-of-the-art SVD-based baselines and structured pruning methods, establishing it as a superior paradigm for efficient LLM compression.