Choose Your Model Size: Any Compression of Large Language Models Without Re-Computation

TL;DR

This paper introduces ACIP, a novel post-training compression method for large language models that enables flexible size reduction without re-computation, balancing efficiency and performance.

Contribution

ACIP is a new algorithm that determines optimal compression-performance trade-offs from a single training run, using iterative pruning and SVD reparametrization.

Findings

Achieves state-of-the-art compression results on open-weight LLMs.

Allows compression to any target size without re-computation.

Complements existing quantization techniques.

Abstract

The adoption of Foundation Models in resource-constrained environments remains challenging due to their large size and inference costs. A promising way to overcome these limitations is post-training compression, which aims to balance reduced model size against performance degradation. This work presents Any Compression via Iterative Pruning (ACIP), a novel algorithmic approach to determine a compression-performance trade-off from a single stochastic gradient descent run. To achieve parameter efficiency, we use an SVD-reparametrization of linear layers and iteratively prune their singular values with a sparsity-inducing penalty. Importantly, the pruning order of the parameters is used to derive a global score map that allows compressing a model to any target size without re-computation. We evaluate ACIP on a large selection of open-weight LLMs and downstream tasks, demonstrating state-of-the-art results compared to existing factorization-based compression methods. We also show that ACIP seamlessly complements common quantization-based compression techniques.