Capability-Guided Compression: Toward Interpretability-Aware Budget Allocation for Large Language Models

TL;DR

This paper introduces Capability-Guided Compression (CGC), a novel framework that allocates compression budgets based on model component capabilities, improving interpretability and addressing phase transition issues in large language model compression.

Contribution

It proposes a capability density measure derived from autoencoder features, providing a new pre-compression predictor for component phase transitions and enabling interpretability-aware compression.

Findings

Capability density is independent of importance scores.

Components with higher capability density reach phase transitions at lower compression ratios.

Theoretical proof links capability density to structural redundancy and phase transition points.

Abstract

Large language model compression has made substantial progress through pruning, quantization, and low-rank decomposition, yet a fundamental limitation persists across all existing methods: compression budgets are allocated without any representation of what individual model components functionally encode. We term this the capability-blind compression problem and argue it is a root cause of two well-documented failures -- the insensitivity of perplexity-based evaluation to reasoning capability loss, and the abrupt phase transitions in model performance recently characterized by Ma et al. (2026). We propose Capability-Guided Compression (CGC), a framework that addresses this by using Sparse Autoencoder (SAE)-derived capability density maps to allocate differential compression budgets across transformer components. Capability density is a formally defined scalar measure combining the feature breadth, activation entropy, and cross-input consistency of a component's SAE feature activation distribution. We prove theoretically that components with higher capability density exhibit lower structural redundancy and reach their individual phase transition points at lower compression ratios, providing the first pre-compression mechanism for component-level phase transition prediction. Experiments on GPT-2 Medium confirm that capability density is statistically independent of Wanda importance scores (Spearman rho = -0.054, n = 384 heads), establishing it as a genuinely novel compression signal orthogonal to all existing importance metrics. We report a negative result on PPL-based compression comparison and provide a principled diagnosis identifying GPT-2 Medium as an insufficient test bed for the full CGC hypothesis. The theoretical framework, density formalism, and orthogonality finding constitute a foundation for capability-aware compression research.