# A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamics

## TL;DR

This paper analyzes the training dynamics and generalization properties of two-layer neural networks and random feature models under gradient descent, revealing exponential convergence and kernel-like behavior.

## Contribution

It provides a comprehensive theoretical analysis of gradient descent dynamics for two-layer neural networks, including convergence rates and generalization error estimates.

## Key findings

- Gradient descent achieves exponential convergence to zero training loss.
- Neural network functions remain close to kernel methods during training.
- Sharp bounds on generalization error are established for various network widths and data sizes.

## Abstract

A fairly comprehensive analysis is presented for the gradient descent dynamics for training two-layer neural network models in the situation when the parameters in both layers are updated. General initialization schemes as well as general regimes for the network width and training data size are considered. In the over-parametrized regime, it is shown that gradient descent dynamics can achieve zero training loss exponentially fast regardless of the quality of the labels. In addition, it is proved that throughout the training process the functions represented by the neural network model are uniformly close to that of a kernel method. For general values of the network width and training data size, sharp estimates of the generalization error is established for target functions in the appropriate reproducing kernel Hilbert space.