Uniform Approximation with Quadratic Neural Networks

TL;DR

This paper demonstrates that deep neural networks with ReQU activation can uniformly approximate H"older-regular functions efficiently, with the approximation quality depending on the function's smoothness and network size.

Contribution

The paper provides a constructive proof that ReQU neural networks can approximate H"older functions with explicit bounds on neurons and layers, extending to RePU activations.

Findings

ReQU networks approximate H"older functions with \\mathcal{O}(\\epsilon^{-d/2r}) neurons.

Approximation depends on the smoothness parameter r of the target function.

Results generalize to RePU activation functions of order p \\geq 2.

Abstract

In this work, we examine the approximation capabilities of deep neural networks utilizing the Rectified Quadratic Unit (ReQU) activation function, defined as \(\max(0,x)^2\), for approximating H\"older-regular functions with respect to the uniform norm. We constructively prove that deep neural networks with ReQU activation can approximate any function within the \(R\)-ball of \(r\)-H\"older-regular functions (\(\mathcal{H}^{r, R}([-1,1]^d)\)) up to any accuracy \(\epsilon \) with at most \(\mathcal{O}\left(\epsilon^{-d /2r}\right)\) neurons and fixed number of layers. This result highlights that the effectiveness of the approximation depends significantly on the smoothness of the target function and the characteristics of the ReQU activation function. Our proof is based on approximating local Taylor expansions with deep ReQU neural networks, demonstrating their ability to capture the behavior of H\"older-regular functions effectively. Furthermore, the results can be straightforwardly generalized to any Rectified Power Unit (RePU) activation function of the form \(\max(0,x)^p\) for \(p \geq 2\), indicating the broader applicability of our findings within this family of activations.