Approximation Rates for Shallow ReLU$^k$ Neural Networks on Sobolev Spaces via the Radon Transform

TL;DR

This paper establishes nearly optimal approximation rates for shallow ReLU$^k$ neural networks on Sobolev spaces using the Radon transform, extending previous results and highlighting their adaptivity for smoothness up to a certain order.

Contribution

It provides a simple proof of near-optimal approximation rates for shallow ReLU$^k$ networks on Sobolev spaces, generalizing existing results through the Radon transform and discrepancy theory.

Findings

Approximation rates are nearly optimal up to logarithmic factors.

Shallow ReLU$^k$ networks adaptively achieve optimal rates for smoothness up to s = k + (d+1)/2.

Results hold for various cases including q ≤ p, p ≥ 2, and s ≤ k + (d+1)/2.

Abstract

Let $\Omega\subset \mathbb{R}^d$ be a bounded domain. We consider the problem of how efficiently shallow neural networks with the ReLU$^k$ activation function can approximate functions from Sobolev spaces $W^s(L_p(\Omega))$ with error measured in the $L_q(\Omega)$-norm. Utilizing the Radon transform and recent results from discrepancy theory, we provide a simple proof of nearly optimal approximation rates in a variety of cases, including when $q\leq p$, $p\geq 2$, and $s \leq k + (d+1)/2$. The rates we derive are optimal up to logarithmic factors, and significantly generalize existing results. An interesting consequence is that the adaptivity of shallow ReLU$^k$ neural networks enables them to obtain optimal approximation rates for smoothness up to order $s = k + (d+1)/2$, even though they represent piecewise polynomials of fixed degree $k$.