Predicting LLM Compression Degradation from Spectral Statistics

TL;DR

This paper introduces a predictive metric based on spectral statistics that accurately estimates the performance degradation of large language models after low-rank compression, enabling efficient compression decisions.

Contribution

It identifies stable rank and information density as key predictors of compression-induced degradation and validates a new predictive metric with high correlation to actual performance loss.

Findings

The interaction term γ · ρ̄_s predicts accuracy degradation with high correlation.

Stable rank and information density dominate performance degradation.

Theoretical links connect spectral predictors to SVD truncation bounds.

Abstract

Matrix-level low-rank compression is a promising way to reduce the cost of large language models, but running compression and evaluating the resulting models on language tasks can be prohibitively expensive. Can compression-induced degradation be predicted before committing to this compute? We systematically analyze the Qwen3 and Gemma3 model families across four representative low-rank compression methods: vanilla SVD, two ASVD variants, and SVD-LLM. We find that stable rank and information density, measured in bits per parameter, dominate performance degradation. The interaction term $\gamma \cdot \bar{\rho}_s$, defined as compression ratio times stable rank, is a robust predictor of accuracy degradation, achieving leave-one-out cross-validation Pearson correlations of $0.890$ for attention layers and $0.839$ for MLP layers. We provide theoretical intuition for why this predictor succeeds by connecting it to standard SVD truncation bounds and error composition mechanisms in transformer layers. These findings enable a predict-then-compress workflow: compute $\gamma \cdot \bar{\rho}_s$ from weights, estimate degradation, and invest compute only in desirable configurations.