Deep ReLU networks and high-order finite element methods II: Chebyshev emulation

TL;DR

This paper develops efficient ReLU neural network constructions for approximating piecewise polynomial functions using Chebyshev polynomial coefficients, achieving superior expression rates and stability in Sobolev norms, with applications to numerical analysis.

Contribution

It introduces novel ReLU NN surrogates based on Chebyshev coefficients that require fewer neurons and provide explicit bounds, improving upon previous monomial-based constructions.

Findings

ReLU NN emulation bounds are explicit and superior to previous methods.

Chebyshev coefficient computation via inverse FFT is efficient.

Exponential approximation rates for analytic functions with singularities.

Abstract

We show expression rates and stability in Sobolev norms of deep feedforward ReLU neural networks (NNs) in terms of the number of parameters defining the NN for continuous, piecewise polynomial functions, on arbitrary, finite partitions $\mathcal{T}$ of a bounded interval $(a,b)$. Novel constructions of ReLU NN surrogates encoding function approximations in terms of Chebyshev polynomial expansion coefficients are developed which require fewer neurons than previous constructions. Chebyshev coefficients can be computed easily from the values of the function in the Clenshaw--Curtis points using the inverse fast Fourier transform. Bounds on expression rates and stability are obtained that are superior to those of constructions based on ReLU NN emulations of monomials as considered in [Opschoor, Petersen and Schwab, 2020] and [Montanelli, Yang and Du, 2021]. All emulation bounds are explicit in terms of the (arbitrary) partition of the interval, the target emulation accuracy and the polynomial degree in each element of the partition. ReLU NN emulation error estimates are provided for various classes of functions and norms, commonly encountered in numerical analysis. In particular, we show exponential ReLU emulation rate bounds for analytic functions with point singularities and develop an interface between Chebfun approximations and constructive ReLU NN emulations.