A lattice-based approach to the expressivity of deep ReLU neural networks

TL;DR

This paper introduces a new family of high-dimensional CPWL functions that grow exponentially in complexity, demonstrating their computation via deep ReLU networks and exploring their approximation properties and practical relevance in channel coding.

Contribution

It presents a novel class of CPWL functions with exponential complexity and analyzes their computation by deep ReLU networks, linking neural network expressivity to decoding tasks.

Findings

Functions have exponential growth in affine pieces with dimension

ReLU networks with quadratic depth can compute these functions

Shallow networks have limited approximation capabilities for these functions

Abstract

We present new families of continuous piecewise linear (CPWL) functions in Rn having a number of affine pieces growing exponentially in $n$. We show that these functions can be seen as the high-dimensional generalization of the triangle wave function used by Telgarsky in 2016. We prove that they can be computed by ReLU networks with quadratic depth and linear width in the space dimension. We also investigate the approximation error of one of these functions by shallower networks and prove a separation result. The main difference between our functions and other constructions is their practical interest: they arise in the scope of channel coding. Hence, computing such functions amounts to performing a decoding operation.