Better Neural Network Expressivity: Subdividing the Simplex

TL;DR

This paper demonstrates that ReLU neural networks can compute all continuous piecewise linear functions on ^n with fewer layers than previously thought, using novel geometric constructions and polyhedral subdivisions.

Contribution

It disproves a conjecture by showing fewer layers are needed for universal CPWL function representation, introducing new constructions for maximum functions with ReLU networks.

Findings

Fewer layers ( log_3(n)) suffice for universal CPWL representation.

ReLU networks with two hidden layers can exactly compute the maximum of five inputs.

The constructions nearly match known lower bounds, advancing understanding of neural network expressivity.

Abstract

This work studies the expressivity of ReLU neural networks with a focus on their depth. A sequence of previous works showed that $\lceil \log_2(n+1) \rceil$ hidden layers are sufficient to compute all continuous piecewise linear (CPWL) functions on $\mathbb{R}^n$. Hertrich, Basu, Di Summa, and Skutella (NeurIPS'21 / SIDMA'23) conjectured that this result is optimal in the sense that there are CPWL functions on $\mathbb{R}^n$, like the maximum function, that require this depth. We disprove the conjecture and show that $\lceil\log_3(n-1)\rceil+1$ hidden layers are sufficient to compute all CPWL functions on $\mathbb{R}^n$. A key step in the proof is that ReLU neural networks with two hidden layers can exactly represent the maximum function of five inputs. More generally, we show that $\lceil\log_3(n-2)\rceil+1$ hidden layers are sufficient to compute the maximum of $n\geq 4$ numbers. Our constructions almost match the $\lceil\log_3(n)\rceil$ lower bound of Averkov, Hojny, and Merkert (ICLR'25) in the special case of ReLU networks with weights that are decimal fractions. The constructions have a geometric interpretation via polyhedral subdivisions of the simplex into ``easier'' polytopes.