Approximation Capabilities of Feedforward Neural Networks with GELU Activations

TL;DR

This paper establishes error bounds for feedforward neural networks with GELU activations, demonstrating their ability to approximate a range of functions and their derivatives with controlled accuracy and network complexity.

Contribution

It provides the first comprehensive error bounds for GELU-activated networks approximating functions and derivatives, including elementary functions like polynomials, exponential, and reciprocal.

Findings

Error bounds hold for functions and derivatives up to any order.

Networks can approximate elementary functions with bounded derivatives.

Analysis includes network size, weight magnitudes, and behavior at infinity.

Abstract

We derive an approximation error bound that holds simultaneously for a function and all its derivatives up to any prescribed order. The bounds apply to elementary functions, including multivariate polynomials, the exponential function, and the reciprocal function, and are obtained using feedforward neural networks with the Gaussian Error Linear Unit (GELU) activation. In addition, we report the network size, weight magnitudes, and behavior at infinity. Our analysis begins with a constructive approximation of multiplication, where we prove the simultaneous validity of error bounds over domains of increasing size for a given approximator. Leveraging this result, we obtain approximation guarantees for division and the exponential function, ensuring that all higher-order derivatives of the resulting approximators remain globally bounded.