Importance-Guided Basis Selection for Low-Rank Decomposition of Large Language Models

TL;DR

This paper introduces BSI, a principled importance-based basis selection method for low-rank compression of large language models, leveraging second-order loss curvature to improve pruning decisions.

Contribution

BSI provides a theoretically grounded, efficient framework for basis pruning in LLMs using a second-order Taylor expansion and a novel Hessian-diagonal estimator.

Findings

BSI outperforms existing low-rank decomposition methods on reasoning benchmarks.

BSI achieves stronger compression with minimal loss in performance.

Theoretical analysis guarantees estimation accuracy and bounds on loss increase.

Abstract

Low-rank decomposition is a compelling approach for compressing large language models, but its effectiveness hinges on selecting which singular-vector bases to retain for a target task. Existing methods such as Basel adapt singular-value coefficients on downstream data and prune bases with small re-learned magnitudes, a heuristic that can be misaligned with task performance because it ignores the local geometry of the loss landscape. We present Basis Selection with Importance (BSI), a principled low-rank compression framework that ranks and prunes bases by directly estimating the expected loss increase incurred when each basis is removed. BSI derives a derivative-based importance score from a second-order Taylor expansion of the task loss with respect to singular values, combining first-order sensitivity and second-order curvature to quantify pruning impact. To make this criterion practical for LLMs, we develop an efficient Hessian-diagonal estimator by adapting the Hutchinson randomized-probing method to loss curvature with symmetric parameter perturbations. We provide a comprehensive theoretical analysis, including loss-increase bounds under basis pruning, explicit propagation of Hessian-diagonal estimation error into these bounds, variance characterization tied to the Hessian spectrum, high-probability sample-complexity guarantees for achieving a target estimation accuracy, and guidance on perturbation intensity. Extensive experiments on mathematical reasoning benchmarks demonstrate that BSI consistently outperforms state-of-the-art low-rank decomposition baselines, with especially strong improvements under deep compression.