Enhancing Delta Compression in LLMs via SVD-based Quantization Error Minimization

TL;DR

PrinMix introduces a mathematically grounded, SVD-based quantization framework for delta compression in LLMs, optimizing error minimization and outperforming state-of-the-art methods on large models.

Contribution

It models quantization as an optimization problem, derives a key scaling mechanism, and employs ILP for optimal bit allocation, advancing delta compression techniques.

Findings

Outperforms SOTA Delta-CoMe by 22.3% on AIME2024

Achieves 6.1% improvement on GQA benchmark

Effectively reduces storage for 7B LLMs

Abstract

Supervised Fine-Tuning (SFT) empowers Large Language Models (LLMs) with exceptional performance on specialized tasks, but it yields dense, high-dimensional delta parameters that pose severe storage and distribution challenges. Singular Value Decomposition (SVD)-based compression offers a compact representation for such delta parameters, but existing methods adopt heuristic quantization without clarifying underlying mechanisms, leading to poor generalizability. In this work, we propose PrinMix, a rigorous SVD-based framework that models quantization as an optimization problem, grounding the design in mathematical mechanisms. We first theoretically derive quantization error and identify a key singular-value-dominated scaling mechanism, which mathematically proves the necessity of mix-precision quantization. We then model the quantization scheme as a 0/1 Integer Linear Programming (ILP) problem, which yields optimal bit-budget-constrained solutions without empirical assumptions. Furthermore, PrinMix integrates a Reconstruction Target Correction (RTC) method to compensate for errors from the $\mathbf{V}$-then-$\mathbf{U}$ sequential quantization process. Extensive experiments confirm PrinMix performs well: for 7B LLMs, PrinMix outperforms SOTA Delta-CoMe on challenging benchmarks by 22.3% on AIME2024 and 6.1% on GQA.