New Evidence of the Two-Phase Learning Dynamics of Neural Networks

TL;DR

This paper investigates the two-phase learning dynamics of deep neural networks, revealing a critical transition characterized by chaos and stability, and how the neural tangent kernel evolves within this process.

Contribution

It introduces an interval-wise analysis revealing two new phenomena, the Chaos Effect and the Cone Effect, that elucidate the transition from exploration to refinement in neural network training.

Findings

Identification of a critical period with high sensitivity to initial conditions.

Discovery that the neural tangent kernel becomes confined to a narrow cone after transition.

Evidence of a two-phase transition in neural network training dynamics.

Abstract

Understanding how deep neural networks learn remains a fundamental challenge in modern machine learning. A growing body of evidence suggests that training dynamics undergo a distinct phase transition, yet our understanding of this transition is still incomplete. In this paper, we introduce an interval-wise perspective that compares network states across a time window, revealing two new phenomena that illuminate the two-phase nature of deep learning. i) \textbf{The Chaos Effect.} By injecting an imperceptibly small parameter perturbation at various stages, we show that the response of the network to the perturbation exhibits a transition from chaotic to stable, suggesting there is an early critical period where the network is highly sensitive to initial conditions; ii) \textbf{The Cone Effect.} Tracking the evolution of the empirical Neural Tangent Kernel (eNTK), we find that after this transition point the model's functional trajectory is confined to a narrow cone-shaped subset: while the kernel continues to change, it gets trapped into a tight angular region. Together, these effects provide a structural, dynamical view of how deep networks transition from sensitive exploration to stable refinement during training.