Efficient Approximation to Analytic and $L^p$ functions by Height-Augmented ReLU Networks

TL;DR

This paper introduces a three-dimensional height-augmented ReLU network architecture that significantly improves the efficiency of approximating analytic and $L^p$ functions, advancing theoretical understanding and network design.

Contribution

It presents a novel 3D network architecture that achieves better approximation rates for analytic and $L^p$ functions with fewer parameters.

Findings

Enhanced exponential approximation rates for analytic functions.

First non-asymptotic high-order approximation for $L^p$ functions.

Provides a theoretically grounded approach for efficient neural network design.

Abstract

This work addresses two fundamental limitations in neural network approximation theory. We demonstrate that a three-dimensional network architecture enables a significantly more efficient representation of sawtooth functions, which serves as the cornerstone in the approximation of analytic and $L^p$ functions. First, we establish substantially improved exponential approximation rates for several important classes of analytic functions and offer a parameter-efficient network design. Second, for the first time, we derive a quantitative and non-asymptotic approximation of high orders for general $L^p$ functions. Our techniques advance the theoretical understanding of the neural network approximation in fundamental function spaces and offer a theoretically grounded pathway for designing more parameter-efficient networks.