IMPACT: Importance-Aware Activation Space Reconstruction

TL;DR

IMPACT is a novel importance-aware activation space reconstruction method that optimizes low-rank compression of large language models by considering activation importance to preserve accuracy.

Contribution

It introduces an importance-weighted activation covariance approach for low-rank compression, improving size reduction while maintaining model performance.

Findings

Achieves up to 55.4% greater model size reduction.

Maintains or improves accuracy compared to state-of-the-art baselines.

Effective across multiple models and tasks.

Abstract

Large language models (LLMs) achieve strong performance across diverse domains but remain difficult to deploy in resource-constrained environments due to their size. Low-rank compression is a common remedy, typically minimizing weight reconstruction error under the assumption that weights are low-rank. However, this assumption often does not hold in LLMs. In contrast, LLM activations exhibit a more pronounced low-rank structure, motivating approaches that minimize activation reconstruction error. This shift alone, however, is not sufficient: different activation dimensions contribute unequally to model performance, and treating them uniformly can lead to accuracy loss. We introduce IMPACT, an importance-aware activation reconstruction framework that links compression to its effect on model performance. IMPACT formulates compression as an optimization problem that integrates activation structure with gradient-based importance, deriving a closed-form solution where reconstruction bases arise from an importance-weighted activation covariance matrix. This yields low-rank compression explicitly optimized for accuracy preservation. Experiments across multiple models and tasks demonstrate that IMPACT achieves up to 55.4% greater model size reduction while maintaining accuracy comparable to or better than state-of-the-art baselines.