Logarithmic Width Suffices for Robust Memorization

TL;DR

This paper investigates the capacity of ReLU neural networks to memorize data robustly against adversarial perturbations, showing that a logarithmic width in the number of samples suffices for robustness.

Contribution

It establishes that logarithmic width in the number of samples is both necessary and sufficient for robust memorization in neural networks.

Findings

Logarithmic width suffices for robust memorization.

Width independent of sample size can achieve robustness.

Bounds on robustness radius for general l_p norms.

Abstract

The memorization capacity of neural networks with a given architecture has been thoroughly studied in many works. Specifically, it is well-known that memorizing $N$ samples can be done using a network of constant width, independent of $N$. However, the required constructions are often quite delicate. In this paper, we consider the natural question of how well feedforward ReLU neural networks can memorize robustly, namely while being able to withstand adversarial perturbations of a given radius. We establish both upper and lower bounds on the possible radius for general $l_p$ norms, implying (among other things) that width logarithmic in the number of input samples is necessary and sufficient to achieve robust memorization (with robustness radius independent of $N$).