Global universality of the two-layer neural network with the $k$-rectified linear unit

TL;DR

This paper proves the universal approximation capability of two-layer neural networks with k-rectified linear units for any domain shape, extending previous results for k=1 and applying to k-sigmoidal functions.

Contribution

It establishes the global universality of two-layer neural networks with k-rectified linear units for all k, generalizing prior work and including k-sigmoidal functions as an application.

Findings

Proves universal approximation for all k-rectified linear units

Extends previous results from k=1 to all k

Includes applications to k-sigmoidal functions

Abstract

This paper concerns the universality of the two-layer neural network with the $k$-rectified linear unit activation function with $k=1,2,\ldots$ with a suitable norm without any restriction on the shape of the domain. This type of result is called global universality, which extends the previous result for $k=1$ by the present authors. This paper covers $k$-sigmoidal functions as an application of the fundamental result on $k$-rectified linear unit functions.