On the Depth of Monotone ReLU Neural Networks and ICNNs

TL;DR

This paper investigates the expressivity of monotone ReLU and input convex neural networks, establishing lower bounds on their depth for computing the maximum function and demonstrating depth separations between these models.

Contribution

It provides new theoretical lower bounds on the depth complexity of ReLU$^+$ and ICNNs for the maximum function, and shows depth separation results between the two models.

Findings

ReLU$^+$ networks cannot approximate MAX$_n$

ICNNs require depth at least n to compute MAX$_n$

Depth-2 ReLU networks cannot be simulated by shallow ICNNs

Abstract

We study two models of ReLU neural networks: monotone networks (ReLU$^+$) and input convex neural networks (ICNN). Our focus is on expressivity, mostly in terms of depth, and we prove the following lower bounds. For the maximum function MAX$_n$ computing the maximum of $n$ real numbers, we show that ReLU$^+$ networks cannot compute MAX$_n$, or even approximate it. We prove a sharp $n$ lower bound on the ICNN depth complexity of MAX$_n$. We also prove depth separations between ReLU networks and ICNNs; for every $k$, there is a depth-2 ReLU network of size $O(k^2)$ that cannot be simulated by a depth-$k$ ICNN. The proofs are based on deep connections between neural networks and polyhedral geometry, and also use isoperimetric properties of triangulations.