Schauder Bases for $C[0, 1]$ Using ReLU, Softplus and Two Sigmoidal Functions

TL;DR

This paper constructs four Schauder bases for the space of continuous functions on [0,1] using ReLU, Softplus, and sigmoidal functions, demonstrating their basis properties and approximation capabilities.

Contribution

It introduces the first Schauder bases for $C[0,1]$ using these activation functions and improves understanding of their approximation properties.

Findings

Established Schauder bases using ReLU, Softplus, and sigmoidal functions.

Proved an $O(1/n)$ approximation bound for the ReLU basis.

Presented a negative result on multivariate function approximation with finite ReLU combinations.

Abstract

We construct four Schauder bases for the space $C[0,1]$, one using ReLU functions, another using Softplus functions, and two more using sigmoidal versions of the ReLU and Softplus functions. This establishes the existence of a basis using these functions for the first time, and improves on the universal approximation property associated with them. We also show an $O(\frac{1}{n})$ approximation bound based on our ReLU basis, and a negative result on constructing multivariate functions using finite combinations of ReLU functions.