Approximation and Gradient Descent Training with Neural Networks

TL;DR

This paper bridges the gap between neural network approximation capabilities and practical training methods by extending theoretical results from gradient flow to gradient descent, especially in over- and under-parametrized regimes.

Contribution

It provides new theoretical guarantees for neural networks trained by gradient descent, aligning approximation theory with practical training scenarios.

Findings

Gradient descent achieves similar approximation bounds as gradient flow.

Over-parametrization guarantees zero training error.

Theoretical results apply to practical training methods.

Abstract

It is well understood that neural networks with carefully hand-picked weights provide powerful function approximation and that they can be successfully trained in over-parametrized regimes. Since over-parametrization ensures zero training error, these two theories are not immediately compatible. Recent work uses the smoothness that is required for approximation results to extend a neural tangent kernel (NTK) optimization argument to an under-parametrized regime and show direct approximation bounds for networks trained by gradient flow. Since gradient flow is only an idealization of a practical method, this paper establishes analogous results for networks trained by gradient descent.