Memorization capacity of deep ReLU neural networks characterized by width and depth

TL;DR

This paper characterizes the memorization capacity of deep ReLU neural networks by establishing optimal bounds on the combined width and depth needed to memorize any dataset with given size and separation, revealing a fundamental trade-off.

Contribution

It provides the first explicit characterization of the width-depth trade-off for memorization capacity, generalizing previous parameter-based bounds and establishing optimality up to logarithmic factors.

Findings

Networks with $W^2L^2=O(N ext{log}(rac{1}{ ext{delta}}))$ can memorize any $N$ samples.

Lower bounds show $W^2L^2= ext{Omega}(N ext{log}(rac{1}{ ext{delta}}))$, proving optimality.

Explicit trade-off between width and depth for memorization capacity.

Abstract

This paper studies the memorization capacity of deep neural networks with ReLU activation. Specifically, we investigate the minimal size of such networks to memorize any $N$ data points in the unit ball with pairwise separation distance $\delta$ and discrete labels. Most prior studies characterize the memorization capacity by the number of parameters or neurons. We generalize these results by constructing neural networks, whose width $W$ and depth $L$ satisfy $W^2L^2= \mathcal{O}(N\log(\delta^{-1}))$, that can memorize any $N$ data samples. We also prove that any such networks should also satisfy the lower bound $W^2L^2=\Omega (N \log(\delta^{-1}))$, which implies that our construction is optimal up to logarithmic factors when $\delta^{-1}$ is polynomial in $N$. Hence, we explicitly characterize the trade-off between width and depth for the memorization capacity of deep neural networks in this regime.