Description of the Training Process of Neural Networks via Ergodic Theorem : Ghost nodes

TL;DR

This paper introduces a novel ergodic-theoretic framework for understanding neural network training, proposing ghost nodes to enhance early training efficiency without affecting long-term performance.

Contribution

It presents a unified ergodic perspective on training dynamics and introduces ghost nodes as a new architectural extension to accelerate early training phases.

Findings

Ghost nodes improve early training convergence.

The Lyapunov exponent diagnostic distinguishes convergence from saddle points.

Ghost dimensions collapse after convergence, preserving original model behavior.

Abstract

Recent studies have proposed interpreting the training process from an ergodic perspective. Building on this foundation, we present a unified framework for understanding and accelerating the training of deep neural networks via stochastic gradient descent (SGD). By analyzing the geometric landscape of the objective function we introduce a practical diagnostic, the running estimate of the largest Lyapunov exponent, which provably distinguishes genuine convergence toward stable minimizers from mere statistical stabilization near saddle points. We then propose a ghost category extension for standard classifiers that adds auxiliary ghost output nodes so the model gains extra descent directions that open a lateral corridor around narrow loss barriers and enable the optimizer to bypass poor basins during the early training phase. We show that this extension strictly reduces the approximation error and that after sufficient convergence the ghost dimensions collapse so that the extended model coincides with the original one and there exists a path in the enlarged parameter space along which the total loss does not increase. Taken together, these results provide a principled architecture level intervention that accelerates early stage trainability while preserving asymptotic behavior and simultaneously serves as an architecture-friendly regularizer.