Self-Abstraction Learning for Effective and Stable Training of Deep Neural Networks

TL;DR

Self-Abstraction Learning (SAL) is a hierarchical training framework that improves the stability and effectiveness of deep neural network training by guiding complex networks with simpler ones in a top-down manner.

Contribution

Introduces SAL, a novel hierarchical training method that enhances stability and generalization of deep networks through sequential guidance from simpler to complex models.

Findings

SAL outperforms conventional training methods across various architectures.

SAL ensures robust generalization in data-scarce and complex network scenarios.

Hierarchical guidance mitigates optimization issues like vanishing gradients.

Abstract

Training large-scale deep neural networks effectively and stably is essential for applying deep learning across various fields. However, conventional methods, which rely on training a single large network, often encounter challenges such as gradient vanishing, overfitting and unstable learning. To overcome these limitations, we introduce Self-Abstraction Learning (SAL), a hierarchical framework. In SAL, networks are arranged by structural complexity, where the simplest topmost network is trained first and its hidden and output layers serve as guidance for the successively more complex networks below. This top-down sequential guidance effectively mitigates optimization issues, enabling stable training of deep architectures. Various experiments across MLP, CNN, and RNN architectures demonstrate that SAL consistently outperforms conventional methods, ensuring robust generalization even in data-scarce and complex network regimes.