Experiments on Properties of Hidden Structures of Sparse Neural Networks

TL;DR

This paper investigates the properties of sparse neural network structures, exploring how sparsity can be induced and its effects on performance, with experiments on various pruning methods, network types, and architecture predictions.

Contribution

It introduces novel methods for inducing sparsity, including prior initialization and pruning techniques, and applies network theory priors to RNNs, alongside performance prediction during neural architecture search.

Findings

Magnitude class blinded pruning achieves 97.5% accuracy on MNIST with 80% compression.

Magnitude class uniform pruning performs significantly worse than blinded pruning.

Performance prediction models for RNNs achieve an R^2 of up to 0.81 using structural information.

Abstract

Sparsity in the structure of Neural Networks can lead to less energy consumption, less memory usage, faster computation times on convenient hardware, and automated machine learning. If sparsity gives rise to certain kinds of structure, it can explain automatically obtained features during learning. We provide insights into experiments in which we show how sparsity can be achieved through prior initialization, pruning, and during learning, and answer questions on the relationship between the structure of Neural Networks and their performance. This includes the first work of inducing priors from network theory into Recurrent Neural Networks and an architectural performance prediction during a Neural Architecture Search. Within our experiments, we show how magnitude class blinded pruning achieves 97.5% on MNIST with 80% compression and re-training, which is 0.5 points more than without compression, that magnitude class uniform pruning is significantly inferior to it and how a genetic search enhanced with performance prediction achieves 82.4% on CIFAR10. Further, performance prediction for Recurrent Networks learning the Reber grammar shows an $R^2$ of up to 0.81 given only structural information.