Pruning's Effect on Generalization Through the Lens of Training and Regularization

TL;DR

This paper empirically investigates how pruning improves model generalization, revealing that factors beyond size reduction, such as enhanced training and regularization effects, are crucial for understanding pruning's benefits.

Contribution

The study challenges the size reduction hypothesis by showing pruning's benefits stem from improved training and regularization, not just smaller models.

Findings

Pruning improves training loss at specific sparsities.

Pruning provides additional regularization, reducing noise impact.

Both training enhancement and regularization are key to pruning's effectiveness.

Abstract

Practitioners frequently observe that pruning improves model generalization. A long-standing hypothesis based on bias-variance trade-off attributes this generalization improvement to model size reduction. However, recent studies on over-parameterization characterize a new model size regime, in which larger models achieve better generalization. Pruning models in this over-parameterized regime leads to a contradiction -- while theory predicts that reducing model size harms generalization, pruning to a range of sparsities nonetheless improves it. Motivated by this contradiction, we re-examine pruning's effect on generalization empirically. We show that size reduction cannot fully account for the generalization-improving effect of standard pruning algorithms. Instead, we find that pruning leads to better training at specific sparsities, improving the training loss over the dense model. We find that pruning also leads to additional regularization at other sparsities, reducing the accuracy degradation due to noisy examples over the dense model. Pruning extends model training time and reduces model size. These two factors improve training and add regularization respectively. We empirically demonstrate that both factors are essential to fully explaining pruning's impact on generalization.