Efficient compression of neural networks and datasets

TL;DR

This paper advances neural network and dataset compression by refining pruning techniques based on MDL principles, leading to better generalization and sample efficiency.

Contribution

It introduces improved sparse optimization methods for model compression, connecting MDL theory with practical pruning, and empirically verifies better generalization.

Findings

Refined pruning methods outperform previous approaches.

Compressed models maintain accuracy while significantly reducing size.

Compressed models exhibit improved sample efficiency and generalization.

Abstract

Compression and generalization are fundamentally related through Solomonoff induction and the minimum description length principle (MDL), which predict that simpler models generalize better when data arises from low-complexity distributions. In this article, we combine insights from algorithmic information theory and techniques from neural network pruning to improve model generalization by identifying the most effective data compression method. Since exact MDL optimization is intractable, we cast it as $\ell_0$ regularized learning and explain why parameter sparsity provides an effective computable approximation of model description length. To identify the best practical approach, we systematically compare and refine complementary sparse optimization methods. In particular, we improve probabilistic pruning through a procedure that does not require Monte Carlo sampling and refine smooth $\ell_0$ approximations with a binary search routine that reduces hyperparameter complexity. Across convolutional networks and transformers evaluated on image and text datasets, our refined methods improve upon their predecessors, achieve substantial model compression with minimal accuracy loss, and yield short data description lengths. Finally, we use these methods in a controlled teacher-student setting to empirically verify the prediction of Solomonoff induction that compressed models learn more sample-efficiently and generalize better.