Splitting Steepest Descent for Growing Neural Architectures

TL;DR

This paper introduces a progressive neural network growth method using a functional steepest descent approach, enabling adaptive splitting of neurons to improve training efficiency and escape saddle points.

Contribution

It proposes a novel neuron splitting criterion and gradient update based on second-order functional steepest descent, advancing neural architecture optimization.

Findings

Effective in escaping saddle points in training.

Enables resource-efficient neural architecture growth.

Provides a theoretical foundation for adaptive network splitting.

Abstract

We develop a progressive training approach for neural networks which adaptively grows the network structure by splitting existing neurons to multiple off-springs. By leveraging a functional steepest descent idea, we derive a simple criterion for deciding the best subset of neurons to split and a splitting gradient for optimally updating the off-springs. Theoretically, our splitting strategy is a second-order functional steepest descent for escaping saddle points in an $\infty$-Wasserstein metric space, on which the standard parametric gradient descent is a first-order steepest descent. Our method provides a new computationally efficient approach for optimizing neural network structures, especially for learning lightweight neural architectures in resource-constrained settings.