Growing Networks with Autonomous Pruning

TL;DR

GNAP is a novel neural network training method that dynamically grows and prunes the network during training, resulting in highly sparse yet accurate models for image classification.

Contribution

This paper introduces GNAP, a method that autonomously grows and prunes neural networks during training to optimize parameter efficiency and accuracy.

Findings

Achieved 99.44% accuracy on MNIST with 6.2k parameters

Achieved 92.2% accuracy on CIFAR10 with 157.8k parameters

Produced highly sparse networks with competitive performance

Abstract

This paper introduces Growing Networks with Autonomous Pruning (GNAP) for image classification. Unlike traditional convolutional neural networks, GNAP change their size, as well as the number of parameters they are using, during training, in order to best fit the data while trying to use as few parameters as possible. This is achieved through two complementary mechanisms: growth and pruning. GNAP start with few parameters, but their size is expanded periodically during training to add more expressive power each time the network has converged to a saturation point. Between these growing phases, model parameters are trained for classification and pruned simultaneously, with complete autonomy by gradient descent. Growing phases allow GNAP to improve their classification performance, while autonomous pruning allows them to keep as few parameters as possible. Experimental results on several image classification benchmarks show that our approach can train extremely sparse neural networks with high accuracy. For example, on MNIST, we achieved 99.44% accuracy with as few as 6.2k parameters, while on CIFAR10, we achieved 92.2\ accuracy with 157.8k parameters.