Piecewise Linear Functions Representable with Infinite Width Shallow ReLU Neural Networks

TL;DR

This paper proves that any continuous piecewise linear function representable by an infinite width shallow ReLU neural network can also be represented by a finite width shallow ReLU network, clarifying the expressive power of such models.

Contribution

It confirms a conjecture that all continuous piecewise linear functions from infinite width shallow ReLU networks are also realizable with finite width networks.

Findings

Proves the equivalence of infinite and finite width representations for continuous piecewise linear functions.

Maps measures on parameter space to measures on the projective sphere to analyze network representations.

Validates a conjecture by Ongie et al. regarding the expressiveness of shallow ReLU networks.

Abstract

This paper analyzes representations of continuous piecewise linear functions with infinite width, finite cost shallow neural networks using the rectified linear unit (ReLU) as an activation function. Through its integral representation, a shallow neural network can be identified by the corresponding signed, finite measure on an appropriate parameter space. We map these measures on the parameter space to measures on the projective $n$-sphere cross $\mathbb{R}$, allowing points in the parameter space to be bijectively mapped to hyperplanes in the domain of the function. We prove a conjecture of Ongie et al. that every continuous piecewise linear function expressible with this kind of infinite width neural network is expressible as a finite width shallow ReLU neural network.