Self-Expanding Neural Networks

TL;DR

This paper introduces a self-expanding neural network approach that adaptively increases capacity during training, reducing the need for architecture redesigns and improving performance on tasks with uncertain architecture requirements.

Contribution

It proposes a natural gradient-based method for dynamically expanding neural networks during training, with theoretical bounds and practical benefits demonstrated.

Findings

Adaptive expansion improves training efficiency.

Method benefits tasks with uncertain architecture size.

Theoretical bounds support the expansion strategy.

Abstract

The results of training a neural network are heavily dependent on the architecture chosen; and even a modification of only its size, however small, typically involves restarting the training process. In contrast to this, we begin training with a small architecture, only increase its capacity as necessary for the problem, and avoid interfering with previous optimization while doing so. We thereby introduce a natural gradient based approach which intuitively expands both the width and depth of a neural network when this is likely to substantially reduce the hypothetical converged training loss. We prove an upper bound on the ``rate'' at which neurons are added, and a computationally cheap lower bound on the expansion score. We illustrate the benefits of such Self-Expanding Neural Networks with full connectivity and convolutions in both classification and regression problems, including those where the appropriate architecture size is substantially uncertain a priori.