Small Contributions, Small Networks: Efficient Neural Network Pruning Based on Relative Importance

TL;DR

This paper presents a novel neural network pruning method based on activation statistics and information theory, which effectively reduces model size while maintaining performance, and introduces a pruning-aware training strategy.

Contribution

The paper introduces an interpretable pruning technique using activation statistics and a regularized training strategy to improve pruning effectiveness.

Findings

Outperforms baseline pruning methods on multiple datasets.

Maintains high accuracy with significantly reduced model size.

Provides an interpretable and statistically grounded pruning approach.

Abstract

Recent advancements have scaled neural networks to unprecedented sizes, achieving remarkable performance across a wide range of tasks. However, deploying these large-scale models on resource-constrained devices poses significant challenges due to substantial storage and computational requirements. Neural network pruning has emerged as an effective technique to mitigate these limitations by reducing model size and complexity. In this paper, we introduce an intuitive and interpretable pruning method based on activation statistics, rooted in information theory and statistical analysis. Our approach leverages the statistical properties of neuron activations to identify and remove weights with minimal contributions to neuron outputs. Specifically, we build a distribution of weight contributions across the dataset and utilize its parameters to guide the pruning process. Furthermore, we propose a Pruning-aware Training strategy that incorporates an additional regularization term to enhance the effectiveness of our pruning method. Extensive experiments on multiple datasets and network architectures demonstrate that our method consistently outperforms several baseline and state-of-the-art pruning techniques.