Big2Small: A Unifying Neural Network Framework for Model Compression

TL;DR

Big2Small introduces a unifying mathematical framework for model compression and proposes a data-free method that encodes large model weights into compact neural representations, achieving competitive results.

Contribution

The paper develops a measure-theoretic unifying framework for various model compression techniques and introduces Big2Small, a data-free compression method using implicit neural representations.

Findings

Big2Small achieves competitive accuracy and compression ratios.

The framework demonstrates mathematical equivalence among different compression techniques.

Experimental results validate the effectiveness of the proposed method.

Abstract

With the development of foundational models, model compression has become a critical requirement. Various model compression approaches have been proposed such as low-rank decomposition, pruning, quantization, ergodic dynamic systems, and knowledge distillation, which are based on different heuristics. To elevate the field from fragmentation to a principled discipline, we construct a unifying mathematical framework for model compression grounded in measure theory. We further demonstrate that each model compression technique is mathematically equivalent to a neural network subject to a regularization. Building upon this mathematical and structural equivalence, we propose an experimentally-verified data-free model compression framework, termed \textit{Big2Small}, which translates Implicit Neural Representations (INRs) from data domain to the domain of network parameters. \textit{Big2Small} trains compact INRs to encode the weights of larger models and reconstruct the weights during inference. To enhance reconstruction fidelity, we introduce Outlier-Aware Preprocessing to handle extreme weight values and a Frequency-Aware Loss function to preserve high-frequency details. Experiments on image classification and segmentation demonstrate that \textit{Big2Small} achieves competitive accuracy and compression ratios compared to state-of-the-art baselines.