Zero Sum SVD: Balancing Loss Sensitivity for Low Rank LLM Compression

TL;DR

Zero Sum SVD introduces a novel post-training low-rank compression method for large language models that automatically allocates heterogeneous ranks across matrices, improving performance without complex optimization.

Contribution

The paper presents ZS-SVD, a global singular component selection technique using activation whitening and loss estimates, eliminating the need for iterative rank optimization.

Findings

Consistent performance improvements across multiple LLMs and benchmarks.

Automatic heterogeneous rank allocation without optimization.

Effective low-rank compression with minimal loss in accuracy.

Abstract

Advances in large language models have driven strong performance across many tasks, but their memory and compute costs still hinder deployment. SVD-based compression reduces storage and can speed up inference via low-rank factors, yet performance depends on how rank is allocated under a global compression ratio. Prior methods often use homogeneous ranks for similarly sized matrices, despite large differences in loss sensitivity, or rely on expensive iterative pre-truncation optimization to determine per matrix ranks. We propose \textbf{Zero Sum SVD} (\textbf{ZS-SVD}), a post-training method that performs \emph{global} singular component selection using activation whitening and first-order calibration loss estimates in whitened coordinates. \textbf{ZS-SVD} prunes components across the whole model with a \textbf{zero sum} rule that keeps the cumulative predicted loss change near zero, automatically yielding heterogeneous ranks without solving a rank allocation optimization. Motivated by evidence that gradients near pretrained solutions exhibit low rank structure, we also introduce an optional lightweight correction that applies a \textbf{single} projected gradient update after truncation, followed by re-truncation. Extensive experiments across multiple LLM architectures show consistent gains across diverse benchmarks and compression ratios. Code is available at https://github.com/mint-vu/Zero-Sum-SVD