Compression for Better: A General and Stable Lossless Compression Framework

TL;DR

This paper introduces LLC, a theoretical framework for stable, lossless model compression that defines error boundaries to ensure no performance loss, and demonstrates its effectiveness with quantization and decomposition techniques.

Contribution

The paper presents LLC, a novel theoretical framework for lossless model compression that systematically defines error boundaries and guides compression techniques.

Findings

LLC effectively achieves lossless compression across various models.

Quantization reformulated as a grouped knapsack problem improves efficiency.

LLC determines layer-wise rank automatically for low-rank decomposition.

Abstract

This work focus on how to stabilize and lossless model compression, aiming to reduce model complexity and enhance efficiency without sacrificing performance due to compression errors. A key challenge is effectively leveraging compression errors and defining the boundaries for lossless compression to minimize model loss. i.e., compression for better. Currently, there is no systematic approach to determining this error boundary or understanding its specific impact on model performance. We propose a general \textbf{L}oss\textbf{L}ess \textbf{C}ompression theoretical framework (\textbf{LLC}), which further delineates the compression neighborhood and higher-order analysis boundaries through the total differential, thereby specifying the error range within which a model can be compressed without loss. To verify the effectiveness of LLC, we apply various compression techniques, including quantization and decomposition. Specifically, for quantization, we reformulate the classic quantization search problem as a grouped knapsack problem within the lossless neighborhood, achieving lossless quantization while improving computational efficiency. For decomposition, LLC addresses the approximation problem under low-rank constraints, automatically determining the rank for each layer and producing lossless low-rank models. We conduct extensive experiments on multiple neural network architectures on different datasets. The results show that without fancy tricks, LLC can effectively achieve lossless model compression. Our code will be made publicly.