A Quantitative Functional Central Limit Theorem for Shallow Neural Networks

TL;DR

This paper establishes a quantitative functional central limit theorem for shallow neural networks, showing convergence rates depend on activation function smoothness, with implications for understanding neural network behavior.

Contribution

It introduces a new quantitative CLT for shallow neural networks with generic activations, linking convergence rates to activation smoothness.

Findings

Logarithmic convergence rate for non-differentiable activations like ReLU

Square root convergence rate for very smooth activation functions

Utilizes advanced Stein-Malliavin techniques for proofs

Abstract

We prove a Quantitative Functional Central Limit Theorem for one-hidden-layer neural networks with generic activation function. The rates of convergence that we establish depend heavily on the smoothness of the activation function, and they range from logarithmic in non-differentiable cases such as the Relu to $\sqrt{n}$ for very regular activations. Our main tools are functional versions of the Stein-Malliavin approach; in particular, we exploit heavily a quantitative functional central limit theorem which has been recently established by Bourguin and Campese (2020).