Adaptive Two-Layer ReLU Neural Network: I. Best Least-squares Approximation

TL;DR

This paper presents an adaptive method for constructing two-layer ReLU neural networks that achieve optimal least-squares approximation of functions, with improved initialization and accuracy, demonstrated on functions with complex features.

Contribution

The paper introduces the ANE method for adaptive neuron enhancement, enabling optimal approximation and better initialization for two-layer ReLU networks.

Findings

Achieves prescribed approximation accuracy for complex functions.

Provides a natural initialization process for training neural networks.

Demonstrates effectiveness on functions with singularities and sharp layers.

Abstract

In this paper, we introduce adaptive neuron enhancement (ANE) method for the best least-squares approximation using two-layer ReLU neural networks (NNs). For a given function f(x), the ANE method generates a two-layer ReLU NN and a numerical integration mesh such that the approximation accuracy is within the prescribed tolerance. The ANE method provides a natural process for obtaining a good initialization which is crucial for training nonlinear optimization problems. Numerical results of the ANE method are presented for functions of two variables exhibiting either intersecting interface singularities or sharp interior layers.