The Cost of Robustness: Tighter Bounds on Parameter Complexity for Robust Memorization in ReLU Nets

TL;DR

This paper analyzes the number of parameters needed for ReLU neural networks to memorize data robustly, providing tighter bounds across different robustness levels and revealing how parameter complexity varies with robustness.

Contribution

It offers a detailed analysis of parameter complexity bounds for robust memorization in ReLU networks across the entire robustness ratio range, improving upon prior results.

Findings

Parameter complexity matches non-robust memorization for small robustness ratios.

Parameter complexity increases with higher robustness ratios.

Tighter bounds are established compared to previous work.

Abstract

We study the parameter complexity of robust memorization for $\mathrm{ReLU}$ networks: the number of parameters required to interpolate any given dataset with $\epsilon$-separation between differently labeled points, while ensuring predictions remain consistent within a $\mu$-ball around each training sample. We establish upper and lower bounds on the parameter count as a function of the robustness ratio $\rho = \mu / \epsilon$. Unlike prior work, we provide a fine-grained analysis across the entire range $\rho \in (0,1)$ and obtain tighter upper and lower bounds that improve upon existing results. Our findings reveal that the parameter complexity of robust memorization matches that of non-robust memorization when $\rho$ is small, but grows with increasing $\rho$.