Theoretical Compression Bounds for Wide Multilayer Perceptrons

TL;DR

This paper provides a theoretical analysis of pruning and quantization in wide neural networks, demonstrating the existence of highly compressed subnetworks with competitive performance through a randomized greedy algorithm.

Contribution

It introduces a rigorous theoretical framework for pruning and quantization in wide MLPs and CNNs, explaining their empirical success without data assumptions.

Findings

Existence of pruned/quantized subnetworks with competitive performance

Extension of results to structured pruning of CNNs

Tradeoff between compressibility and network width

Abstract

Pruning and quantization techniques have been broadly successful in reducing the number of parameters needed for large neural networks, yet theoretical justification for their empirical success falls short. We consider a randomized greedy compression algorithm for pruning and quantization post-training and use it to rigorously show the existence of pruned/quantized subnetworks of multilayer perceptrons (MLPs) with competitive performance. We further extend our results to structured pruning of MLPs and convolutional neural networks (CNNs), thus providing a unified analysis of pruning in wide networks. Our results are free of data assumptions, and showcase a tradeoff between compressibility and network width. The algorithm we consider bears some similarities with Optimal Brain Damage (OBD) and can be viewed as a post-training randomized version of it. The theoretical results we derive bridge the gap between theory and application for pruning/quantization, and provide a justification for the empirical success of compression in wide multilayer perceptrons.