LION-DG: Layer-Informed Initialization with Deep Gradient Protocols for Accelerated Neural Network Training

TL;DR

LION-DG introduces a layer-informed initialization method for deep neural networks that stabilizes early training, accelerates convergence, and requires no hyperparameters, demonstrated on CIFAR datasets with DenseNet-DS and ResNet-DS.

Contribution

It proposes a novel layer-informed initialization scheme that implements Gradient Awakening, improving training speed and stability without additional hyperparameters or computational cost.

Findings

DenseNet-DS: +8.3% faster convergence on CIFAR-10

Hybrid LSUV + LION-DG achieves 81.92% accuracy on CIFAR-10

ResNet-DS: +11.3% speedup on CIFAR-100

Abstract

Weight initialization remains decisive for neural network optimization, yet existing methods are largely layer-agnostic. We study initialization for deeply-supervised architectures with auxiliary classifiers, where untrained auxiliary heads can destabilize early training through gradient interference. We propose LION-DG, a layer-informed initialization that zero-initializes auxiliary classifier heads while applying standard He-initialization to the backbone. We prove that this implements Gradient Awakening: auxiliary gradients are exactly zero at initialization, then phase in naturally as weights grow -- providing an implicit warmup without hyperparameters. Experiments on CIFAR-10 and CIFAR-100 with DenseNet-DS and ResNet-DS architectures demonstrate: (1) DenseNet-DS: +8.3% faster convergence on CIFAR-10 with comparable accuracy, (2) Hybrid approach: Combining LSUV with LION-DG achieves best accuracy (81.92% on CIFAR-10), (3) ResNet-DS: Positive speedup on CIFAR-100 (+11.3%) with side-tap auxiliary design. We identify architecture-specific trade-offs and provide clear guidelines for practitioners. LION-DG is simple, requires zero hyperparameters, and adds no computational overhead.