Growing Artificial Neural Networks

TL;DR

This paper introduces Artificial Neurogenesis (ANG), a novel algorithm that grows neural networks during training to enable efficient deployment on low SWaP hardware, maintaining high accuracy with fewer weights.

Contribution

ANG is a new method that grows neural networks based on training data, reducing size while preserving accuracy, unlike traditional pruning approaches.

Findings

ANG achieves comparable accuracy with fewer weights.

ANG reduces network size by approximately 65%.

Experimental results on LeNet-5 demonstrate effectiveness.

Abstract

Pruning is a legitimate method for reducing the size of a neural network to fit in low SWaP hardware, but the networks must be trained and pruned offline. We propose an algorithm, Artificial Neurogenesis (ANG), that grows rather than prunes the network and enables neural networks to be trained and executed in low SWaP embedded hardware. ANG accomplishes this by using the training data to determine critical connections between layers before the actual training takes place. Our experiments use a modified LeNet-5 as a baseline neural network that achieves a test accuracy of 98.74% using a total of 61,160 weights. An ANG grown network achieves a test accuracy of 98.80% with only 21,211 weights.