Mathematical Foundations of Neural Tangents and Infinite-Width Networks

TL;DR

This paper develops a rigorous mathematical framework for understanding neural networks in the infinite-width limit using the Neural Tangent Kernel, introducing a new architecture and analyzing its spectral properties and training behavior.

Contribution

It introduces the NTK-ECRN architecture with Fourier features and stochastic depth, providing theoretical bounds and insights into kernel dynamics and generalization.

Findings

Validated kernel behavior on synthetic and benchmark datasets

Demonstrated improved training stability and generalization

Linked spectral properties to optimization performance

Abstract

We investigate the mathematical foundations of neural networks in the infinite-width regime through the Neural Tangent Kernel (NTK). We propose the NTK-Eigenvalue-Controlled Residual Network (NTK-ECRN), an architecture integrating Fourier feature embeddings, residual connections with layerwise scaling, and stochastic depth to enable rigorous analysis of kernel evolution during training. Our theoretical contributions include deriving bounds on NTK dynamics, characterizing eigenvalue evolution, and linking spectral properties to generalization and optimization stability. Empirical results on synthetic and benchmark datasets validate the predicted kernel behavior and demonstrate improved training stability and generalization. This work provides a comprehensive framework bridging infinite-width theory and practical deep-learning architectures.