Approximation Rates of Shallow Neural Networks: Barron Spaces, Activation Functions and Optimality Analysis

TL;DR

This paper analyzes the approximation capabilities of shallow neural networks with exponential power activation functions, revealing their limitations and optimal rates within Barron and Sobolev spaces, and highlighting the impact of dimension and smoothness.

Contribution

It provides a detailed analysis of approximation rates for ReLU$^{k}$ activations, establishing optimal bounds and clarifying the influence of smoothness and dimensionality on neural network approximation.

Findings

Optimal approximation rates are established for functions in Barron and Sobolev spaces.

ReLU$^{k}$ activations cannot achieve optimal rates under certain coefficient bounds.

The curse of dimensionality is confirmed in the approximation capabilities of shallow neural networks.

Abstract

This paper investigates the approximation properties of shallow neural networks with activation functions that are powers of exponential functions. It focuses on the dependence of the approximation rate on the dimension and the smoothness of the function being approximated within the Barron function space. We examine the approximation rates of ReLU$^{k}$ activation functions, proving that the optimal rate cannot be achieved under $\ell^{1}$-bounded coefficients or insufficient smoothness conditions. We also establish optimal approximation rates in various norms for functions in Barron spaces and Sobolev spaces, confirming the curse of dimensionality. Our results clarify the limits of shallow neural networks' approximation capabilities and offer insights into the selection of activation functions and network structures.