Approximation bounds for norm constrained neural networks with applications to regression and GANs

TL;DR

This paper establishes approximation bounds for norm-constrained ReLU neural networks, analyzing their effectiveness in regression and GANs, and providing convergence rates and optimal learning guarantees.

Contribution

It introduces new upper and lower bounds on approximation errors for norm-constrained neural networks and applies these results to regression and generative adversarial networks.

Findings

Derived approximation bounds for smooth functions.

Established convergence rates for over-parameterized networks.

Showed GANs can achieve optimal distribution learning rates.

Abstract

This paper studies the approximation capacity of ReLU neural networks with norm constraint on the weights. We prove upper and lower bounds on the approximation error of these networks for smooth function classes. The lower bound is derived through the Rademacher complexity of neural networks, which may be of independent interest. We apply these approximation bounds to analyze the convergences of regression using norm constrained neural networks and distribution estimation by GANs. In particular, we obtain convergence rates for over-parameterized neural networks. It is also shown that GANs can achieve optimal rate of learning probability distributions, when the discriminator is a properly chosen norm constrained neural network.