Gradient Descent in Neural Networks as Sequential Learning in RKBS

TL;DR

This paper develops a new theoretical framework for neural network training by representing gradient descent as sequential learning in reproducing kernel Banach spaces, extending beyond the over-parametrized regime.

Contribution

It introduces an exact power-series representation of neural networks in RKBS, enabling analysis of training dynamics beyond the NTK approximation.

Findings

Gradient descent training can be exactly modeled as sequential learning in RKBS.

Provides new bounds on uniform convergence related to iteration count and learning rate.

Extends theoretical understanding of neural network training beyond wide networks.

Abstract

The study of Neural Tangent Kernels (NTKs) has provided much needed insight into convergence and generalization properties of neural networks in the over-parametrized (wide) limit by approximating the network using a first-order Taylor expansion with respect to its weights in the neighborhood of their initialization values. This allows neural network training to be analyzed from the perspective of reproducing kernel Hilbert spaces (RKHS), which is informative in the over-parametrized regime, but a poor approximation for narrower networks as the weights change more during training. Our goal is to extend beyond the limits of NTK toward a more general theory. We construct an exact power-series representation of the neural network in a finite neighborhood of the initial weights as an inner product of two feature maps, respectively from data and weight-step space, to feature space, allowing neural network training to be analyzed from the perspective of reproducing kernel {\em Banach} space (RKBS). We prove that, regardless of width, the training sequence produced by gradient descent can be exactly replicated by regularized sequential learning in RKBS. Using this, we present novel bound on uniform convergence where the iterations count and learning rate play a central role, giving new theoretical insight into neural network training.