Approximation Rates for Neural Networks with General Activation Functions

TL;DR

This paper establishes new approximation rates for neural networks with general activation functions, extending previous results to broader classes and demonstrating improved rates via stratified sampling.

Contribution

It introduces novel approximation rate results for neural networks with general activation functions, including bounded and integrable types, and proposes stratified sampling to enhance approximation.

Findings

Dimension-independent approximation rates for polynomially-decaying activation functions.

Approximation rates for bounded, integrable activation functions without decay assumptions.

Stratified sampling improves approximation rates under mild conditions.

Abstract

We prove some new results concerning the approximation rate of neural networks with general activation functions. Our first result concerns the rate of approximation of a two layer neural network with a polynomially-decaying non-sigmoidal activation function. We extend the dimension independent approximation rates previously obtained to this new class of activation functions. Our second result gives a weaker, but still dimension independent, approximation rate for a larger class of activation functions, removing the polynomial decay assumption. This result applies to any bounded, integrable activation function. Finally, we show that a stratified sampling approach can be used to improve the approximation rate for polynomially decaying activation functions under mild additional assumptions.