Renormalized Sparse Neural Network Pruning

TL;DR

This paper introduces a renormalization technique for pruned neural networks that significantly improves accuracy by ensuring error convergence, supported by theoretical proofs and experiments on standard datasets.

Contribution

It presents a novel renormalization method for sparse neural networks, with theoretical guarantees and empirical validation showing improved accuracy over standard pruning.

Findings

Renormalization improves accuracy of pruned networks.

Error converges to zero with renormalization, not without.

Experimental results on MNIST, Fashion MNIST, CIFAR-10 confirm effectiveness.

Abstract

Large neural networks are heavily over-parameterized. This is done because it improves training to optimality. However once the network is trained, this means many parameters can be zeroed, or pruned, leaving an equivalent sparse neural network. We propose renormalizing sparse neural networks in order to improve accuracy. We prove that our method's error converges to zero as network parameters cluster or concentrate. We prove that without renormalizing, the error does not converge to zero in general. We experiment with our method on real world datasets MNIST, Fashion MNIST, and CIFAR-10 and confirm a large improvement in accuracy with renormalization versus standard pruning.