Low-Rank Prehab: Preparing Neural Networks for SVD Compression

TL;DR

Low-Rank Prehab is a novel fine-tuning approach that prepares neural networks for more effective low-rank SVD compression, reducing accuracy loss and enhancing post-compression performance.

Contribution

We introduce Low-Rank Prehab, a pre-compression fine-tuning method that encourages low-rank structure in weights, improving SVD-based compression outcomes for neural networks.

Findings

Prehab reduces accuracy drop after compression.

Prehab improves post-finetuning performance.

Outperforms state-of-the-art SVD techniques across models.

Abstract

Low-rank approximation methods such as singular value decomposition (SVD) and its variants (e.g., Fisher-weighted SVD, Activation SVD) have recently emerged as effective tools for neural network compression. In this setting, decomposition acts as a "surgical" intervention, followed by fine-tuning that serves as "rehab" to recover accuracy. Inspired by prehabilitation in surgery, we introduce a pre-compression fine-tuning stage, Low-Rank Prehab, that explicitly encourages low-rank structure in weight matrices while preserving task performance. By conditioning the model before SVD, Prehab steers weights toward spectrally compact regions of the parameter space, enabling smoother low-rank approximation and improved recovery. Experiments on large language models (LLMs) and other Transformer-based architectures, including Vision Transformers (ViTs), show that Prehab substantially reduces the immediate accuracy drop after compression and consistently improves post-finetuning performance. Across a wide range of compression ratios, our method outperforms state-of-the-art SVD-based techniques such as SVD-LLM, highlighting the importance of preparing models for compression rather than only improving the compression and recovery stages. Source code is available at https://github.com/niqretnuh/PREHAB-SVD