Complexity of One-Dimensional ReLU DNNs

TL;DR

This paper analyzes the complexity of one-dimensional ReLU deep neural networks, showing how the expected number of linear regions scales with network width and proposing a sparsity measure related to function approximation.

Contribution

It provides a theoretical analysis of the expected linear regions in 1D ReLU networks and introduces a function-adaptive sparsity concept for approximation efficiency.

Findings

Expected linear regions grow linearly with total neurons in all layers.

Derived asymptotic growth of linear regions in infinite-width limit.

Introduced a sparsity measure comparing network regions to approximation needs.

Abstract

We study the expressivity of one-dimensional (1D) ReLU deep neural networks through the lens of their linear regions. For randomly initialized, fully connected 1D ReLU networks (He scaling with nonzero bias) in the infinite-width limit, we prove that the expected number of linear regions grows as $\sum_{i = 1}^L n_i + \mathop{{o}}\left(\sum_{i = 1}^L{n_i}\right) + 1$, where $n_\ell$ denotes the number of neurons in the $\ell$-th hidden layer. We also propose a function-adaptive notion of sparsity that compares the expected regions used by the network to the minimal number needed to approximate a target within a fixed tolerance.