Neural networks with superexpressive activations and integer weights

TL;DR

This paper introduces a novel activation function enabling neural networks with integer weights to approximate continuous functions on high-dimensional domains, providing explicit bounds and convergence rates for regression tasks.

Contribution

It presents a new activation function and analyzes the approximation capabilities and convergence rates of neural networks with integer weights for continuous functions.

Findings

Networks can approximate continuous functions with integer weights.

Derived bounds on integer weights for epsilon-approximation.

Established convergence rate of order n^{-2β/(2β+d)} log n.

Abstract

An example of an activation function $\sigma$ is given such that networks with activations $\{\sigma, \lfloor\cdot\rfloor\}$, integer weights and a fixed architecture depending on $d$ approximate continuous functions on $[0,1]^d$. The range of integer weights required for $\varepsilon$-approximation of H\"older continuous functions is derived, which leads to a convergence rate of order $n^{\frac{-2\beta}{2\beta+d}}\log_2n$ for neural network regression estimation of unknown $\beta$-H\"older continuous function with given $n$ samples.