On the Expressiveness of Rational ReLU Neural Networks With Bounded Depth

TL;DR

This paper investigates the minimum depth required for rational ReLU neural networks to represent a specific function, showing that the depth grows logarithmically with the input size, thus confirming and extending previous conjectures.

Contribution

It establishes new lower bounds on the depth of rational ReLU networks needed to represent the function $F_n$, extending prior results from integer weights to decimal and N-ary fractions.

Findings

Networks with decimal fractional weights need at least $oxed{ ext{log}_3(n+1)}$ layers.

N-ary fractional weights require depth at least proportional to $rac{ ext{ln} n}{ ext{ln} ext{ln} N}$.

Results partially confirm the conjecture for rational weights and provide the first non-constant lower bounds for practical networks.

Abstract

To confirm that the expressive power of ReLU neural networks grows with their depth, the function $F_n = \max \{0,x_1,\ldots,x_n\}$ has been considered in the literature. A conjecture by Hertrich, Basu, Di Summa, and Skutella [NeurIPS 2021] states that any ReLU network that exactly represents $F_n$ has at least $\lceil\log_2 (n+1)\rceil$ hidden layers. The conjecture has recently been confirmed for networks with integer weights by Haase, Hertrich, and Loho [ICLR 2023]. We follow up on this line of research and show that, within ReLU networks whose weights are decimal fractions, $F_n$ can only be represented by networks with at least $\lceil\log_3 (n+1)\rceil$ hidden layers. Moreover, if all weights are $N$-ary fractions, then $F_n$ can only be represented by networks with at least $\Omega( \frac{\ln n}{\ln \ln N})$ layers. These results are a partial confirmation of the above conjecture for rational ReLU networks, and provide the first non-constant lower bound on the depth of practically relevant ReLU networks.