On minimal representations of shallow ReLU networks

TL;DR

This paper characterizes the minimal number of neurons needed for shallow ReLU networks to represent continuous piecewise affine functions, revealing differences between one-dimensional and higher-dimensional cases.

Contribution

It provides a precise characterization of minimal network sizes, describes the structure of minimal networks as a smooth manifold, and offers criteria for hyperplanes to realize all such functions.

Findings

Minimal representations use n, n+1, or n+2 neurons.

In 1D, at most n+1 neurons are needed; higher dimensions may require n+2.

The set of minimal networks forms a smooth manifold with known dimension.

Abstract

The realization function of a shallow ReLU network is a continuous and piecewise affine function $f:\mathbb R^d\to \mathbb R$, where the domain $\mathbb R^{d}$ is partitioned by a set of $n$ hyperplanes into cells on which $f$ is affine. We show that the minimal representation for $f$ uses either $n$, $n+1$ or $n+2$ neurons and we characterize each of the three cases. In the particular case, where the input layer is one-dimensional, minimal representations always use at most $n+1$ neurons but in all higher dimensional settings there are functions for which $n+2$ neurons are needed. Then we show that the set of minimal networks representing $f$ forms a $C^\infty$-submanifold $M$ and we derive the dimension and the number of connected components of $M$. Additionally, we give a criterion for the hyperplanes that guarantees that all continuous, piecewise affine functions are realization functions of appropriate ReLU networks.