Shallow ReLU$^s$ Networks in $L^p$-Type and Sobolev Spaces: Approximation and Path-Norm Controlled Generalization

TL;DR

This paper analyzes the approximation capabilities and generalization bounds of shallow ReLU$^s$ networks in $L^p$-type and Sobolev spaces, providing improved rates and minimax-optimal bounds under path-norm control.

Contribution

It introduces new approximation bounds for shallow ReLU$^s$ networks in $L^p$ and Sobolev spaces, and establishes minimax-optimal generalization bounds for regression tasks.

Findings

Improved approximation rates for shallow ReLU$^s$ networks in $L^p$ spaces.

Derived minimax-optimal generalization bounds for regression with path-norm regularization.

Embedded Sobolev spaces into spectral Barron spaces for approximation analysis.

Abstract

We study approximation by shallow ReLU$^s$ networks, $\sigma_s(t)=\max{0,t}^s$, and the generalization behavior of such networks under $\ell_1$ path-norm control. For the $L^p$-type integral spaces $\widetilde{\mathcal{F}}_{p,\tau_d,s}$, $1\le p\le2$, we establish approximation bounds for shallow networks using spherical harmonic analysis. In particular, when the parameter measure is the uniform measure $\tau_d$ and $p<p^*=(2d+2)/(d+3)$, we obtain the rate $O(m^{-1/2-d(2-p)/(2d(2-p)+2p(2s+d+1))}\log^{3/2}m)$, which improves the corresponding random-feature rate. We also derive approximation rates for Sobolev spaces $W^{\alpha,p}$ in the range $1\le p<2$ by embedding them into spectral Barron spaces. Finally, for nonparametric regression with sub-Gaussian noise, we prove minimax-optimal generalization bounds for path-norm-regularized shallow ReLU$^s$ networks over Barron and Sobolev spaces, with matching lower bounds up to logarithmic factors.