Optimizing Singular Spectrum for Large Language Model Compression

TL;DR

This paper introduces SoCo, a data-driven singular spectrum optimization framework for large language model compression, which adaptively prunes components based on learned importance scores to better preserve performance.

Contribution

SoCo employs a learnable diagonal matrix and a three-stage training process to refine importance scores, enabling adaptive pruning and improved compression of LLMs.

Findings

Outperforms state-of-the-art compression methods on multiple benchmarks.

Effectively balances model size reduction with performance retention.

Adaptive pruning based on learned importance scores enhances compression efficiency.

Abstract

Large language models (LLMs) have demonstrated remarkable capabilities, yet prohibitive parameter complexity often hinders their deployment. Existing singular value decomposition (SVD) based compression methods simply deem singular values as importance scores of decomposed components. However, this importance ordered by singular values does not necessarily correlate with the performance of a downstream task. In this work, we introduce SoCo (Singular spectrum optimization for large language model Compression), a novel compression framework that learns to rescale the decomposed components of SVD in a data-driven manner. Concretely, we employ a learnable diagonal matrix to assign importance scores for singular spectrum and develop a three-stage training process that progressively refines these scores from initial coarse compression to fine-grained sparsification-thereby striking an effective balance between aggressive model compression and performance preservation. Thanks to the learnable singular spectrum, SoCo adaptively prunes components according to the sparsified importance scores, rather than relying on the fixed order of singular values. More importantly, the remaining components with amplified importance scores can compensate for the loss of the pruned ones. Experimental evaluations across multiple LLMs and benchmarks demonstrate that SoCo surpasses the state-of-the-art methods in model compression.