On Enhancing Expressive Power via Compositions of Single Fixed-Size ReLU Network

TL;DR

This paper demonstrates that repeatedly composing a fixed-size ReLU network can significantly enhance its expressive power, enabling approximation of complex functions with fixed-size components, revealing the potential of continuous-depth networks.

Contribution

We prove that compositions of a fixed-size ReLU network can approximate Lipschitz and continuous functions on [0,1]^d, showing the power of fixed-size networks in deep compositions.

Findings

Repetitive compositions approximate 1-Lipschitz functions with error O(r^{-1/d})

Extension to generic continuous functions with modulus of continuity

Fixed-size ReLU networks can generate highly expressive continuous-depth networks

Abstract

This paper explores the expressive power of deep neural networks through the framework of function compositions. We demonstrate that the repeated compositions of a single fixed-size ReLU network exhibit surprising expressive power, despite the limited expressive capabilities of the individual network itself. Specifically, we prove by construction that $\mathcal{L}_2\circ \boldsymbol{g}^{\circ r}\circ \boldsymbol{\mathcal{L}}_1$ can approximate $1$-Lipschitz continuous functions on $[0,1]^d$ with an error $\mathcal{O}(r^{-1/d})$, where $\boldsymbol{g}$ is realized by a fixed-size ReLU network, $\boldsymbol{\mathcal{L}}_1$ and $\mathcal{L}_2$ are two affine linear maps matching the dimensions, and $\boldsymbol{g}^{\circ r}$ denotes the $r$-times composition of $\boldsymbol{g}$. Furthermore, we extend such a result to generic continuous functions on $[0,1]^d$ with the approximation error characterized by the modulus of continuity. Our results reveal that a continuous-depth network generated via a dynamical system has immense approximation power even if its dynamics function is time-independent and realized by a fixed-size ReLU network.