Geometric separation and constructive universal approximation with two hidden layers

TL;DR

This paper presents a geometric method for constructing neural networks with two hidden layers that can universally approximate any continuous function on compact sets, providing explicit constructions and sharp results especially for finite sets.

Contribution

It introduces a geometric construction for neural networks with two hidden layers that guarantees universal approximation, including explicit methods and sharp results for finite sets.

Findings

Networks with two hidden layers can approximate any continuous function on compact sets.

Explicit geometric constructions lead to universal approximation theorems.

Sharp depth-2 approximation results are obtained for finite sets.

Abstract

We give a geometric construction of neural networks that separate disjoint compact subsets of $\Bbb R^n$, and use it to obtain a constructive universal approximation theorem. Specifically, we show that networks with two hidden layers and either a sigmoidal activation (i.e., strictly monotone bounded continuous) or the ReLU activation can approximate any real-valued continuous function on an arbitrary compact set $K\subset\Bbb R^n$ to any prescribed accuracy in the uniform norm. For finite $K$, the construction simplifies and yields a sharp depth-2 (single hidden layer) approximation result.