COMPOT: Calibration-Optimized Matrix Procrustes Orthogonalization for Transformers Compression

TL;DR

COMPOT is a training-free, calibration-based framework for Transformer compression that uses orthogonal dictionaries and a dynamic layer-wise rate allocation to outperform traditional low-rank and sparse methods.

Contribution

It introduces COMPOT, a novel non-iterative, calibration-optimized method employing orthogonal dictionaries and adaptive compression for Transformers.

Findings

Outperforms low-rank and sparse baselines in quality-compression trade-off.

Compatible with post-training quantization for extreme compression.

Effective across diverse architectures and tasks.

Abstract

Post-training compression of Transformer models commonly relies on truncated singular value decomposition (SVD). However, enforcing a single shared subspace can degrade accuracy even at moderate compression. Sparse dictionary learning provides a more flexible union-of-subspaces representation, but existing approaches often suffer from iterative dictionary and coefficient updates. We propose COMPOT (Calibration-Optimized Matrix Procrustes Orthogonalization for Transformers), a training-free compression framework that uses a small calibration dataset to estimate a sparse weight factorization. COMPOT employs orthogonal dictionaries that enable closed-form Procrustes updates for the dictionary and analytical single-step sparse coding for the coefficients, eliminating iterative optimization. To handle heterogeneous layer sensitivity under a global compression budget, COMPOT further introduces a one-shot dynamic allocation strategy that adaptively redistributes layer-wise compression rates. Extensive experiments across diverse architectures and tasks show that COMPOT consistently delivers a superior quality-compression trade-off over strong low-rank and sparse baselines, while remaining fully compatible with post-training quantization for extreme compression. Code is available $\href{https://github.com/mts-ai/COMPOT}{here}$.