$\epsilon$-rank and the Staircase Phenomenon: New Insights into Neural Network Training Dynamics

TL;DR

This paper introduces the $\epsilon$-rank metric to analyze neural network training dynamics, revealing a staircase pattern during loss reduction and proposing a pre-training method to enhance training efficiency and accuracy.

Contribution

The paper defines the $\epsilon$-rank$ as a new metric, proves its negative correlation with loss, and develops a pre-training strategy to improve training outcomes.

Findings

$\epsilon$-rank$ increases during training and correlates with loss reduction.

A staircase pattern is observed in the training process.

Pre-training the initial layer elevates $\epsilon$-rank$ and accelerates training.

Abstract

Understanding the training dynamics of deep neural networks (DNNs), particularly how they evolve low-dimensional features from high-dimensional data, remains a central challenge in deep learning theory. In this work, we introduce the concept of $\epsilon$-rank, a novel metric quantifying the effective feature of neuron functions in the terminal hidden layer. Through extensive experiments across diverse tasks, we observe a universal staircase phenomenon: during training process implemented by the standard stochastic gradient descent methods, the decline of the loss function is accompanied by an increase in the $\epsilon$-rank and exhibits a staircase pattern. Theoretically, we rigorously prove a negative correlation between the loss lower bound and $\epsilon$-rank, demonstrating that a high $\epsilon$-rank is essential for significant loss reduction. Moreover, numerical evidences show that within the same deep neural network, the $\epsilon$-rank of the subsequent hidden layer is higher than that of the previous hidden layer. Based on these observations, to eliminate the staircase phenomenon, we propose a novel pre-training strategy on the initial hidden layer that elevates the $\epsilon$-rank of the terminal hidden layer. Numerical experiments validate its effectiveness in reducing training time and improving accuracy across various tasks. Therefore, the newly introduced concept of $\epsilon$-rank is a computable quantity that serves as an intrinsic effective metric characteristic for deep neural networks, providing a novel perspective for understanding the training dynamics of neural networks and offering a theoretical foundation for designing efficient training strategies in practical applications.