Accelerated Training via Incrementally Growing Neural Networks using Variance Transfer and Learning Rate Adaptation

TL;DR

This paper introduces a novel neural network growth method that stabilizes training dynamics and adapts learning rates, enabling faster training with comparable or better accuracy than fixed-size models.

Contribution

It proposes a dynamic parameterization scheme and learning rate adaptation mechanism to efficiently grow neural networks during training, improving speed without sacrificing accuracy.

Findings

Achieves comparable or better accuracy than fixed-size models.

Reduces training computation and time significantly.

Demonstrates real wall-clock speedups in training.

Abstract

We develop an approach to efficiently grow neural networks, within which parameterization and optimization strategies are designed by considering their effects on the training dynamics. Unlike existing growing methods, which follow simple replication heuristics or utilize auxiliary gradient-based local optimization, we craft a parameterization scheme which dynamically stabilizes weight, activation, and gradient scaling as the architecture evolves, and maintains the inference functionality of the network. To address the optimization difficulty resulting from imbalanced training effort distributed to subnetworks fading in at different growth phases, we propose a learning rate adaption mechanism that rebalances the gradient contribution of these separate subcomponents. Experimental results show that our method achieves comparable or better accuracy than training large fixed-size models, while saving a substantial portion of the original computation budget for training. We demonstrate that these gains translate into real wall-clock training speedups.