Quantization vs Pruning: Insights from the Strong Lottery Ticket Hypothesis

TL;DR

This paper extends the Strong Lottery Ticket Hypothesis to quantized neural networks, providing theoretical insights into how pruning can exactly represent discrete, low-precision models with optimal overparameterization bounds.

Contribution

It develops new theoretical results for the Random Subset Sum Problem in a quantized setting and extends SLTH to finite-precision networks, showing exact representation of target discrete networks.

Findings

Quantized networks can be exactly represented via pruning.

Provides optimal bounds on overparameterization for quantized models.

Extends SLTH framework to finite-precision neural networks.

Abstract

Quantization is an essential technique for making neural networks more efficient, yet our theoretical understanding of it remains limited. Previous works demonstrated that extremely low-precision networks, such as binary networks, can be constructed by pruning large, randomly-initialized networks, and showed that the ratio between the size of the original and the pruned networks is at most polylogarithmic. The specific pruning method they employed inspired a line of theoretical work known as the Strong Lottery Ticket Hypothesis (SLTH), which leverages insights from the Random Subset Sum Problem. However, these results primarily address the continuous setting and cannot be applied to extend SLTH results to the quantized setting. In this work, we build on foundational results by Borgs et al. on the Number Partitioning Problem to derive new theoretical results for the Random Subset Sum Problem in a quantized setting. Using these results, we then extend the SLTH framework to finite-precision networks. While prior work on SLTH showed that pruning allows approximation of a certain class of neural networks, we demonstrate that, in the quantized setting, the analogous class of target discrete neural networks can be represented exactly, and we prove optimal bounds on the necessary overparameterization of the initial network as a function of the precision of the target network.