Robustness in deep learning: The good (width), the bad (depth), and the ugly (initialization)

TL;DR

This paper investigates how width, depth, initialization, and training regimes affect the robustness of deep neural networks, revealing that over-parameterization and specific initializations can improve robustness, while others may harm it.

Contribution

The paper provides a theoretical analysis of robustness in deep networks, highlighting the effects of width, depth, and initialization across different training regimes, extending prior work.

Findings

Width improves robustness in over-parameterized networks.

Depth's effect on robustness depends on initialization and training mode.

Non-lazy training benefits from increased width for robustness.

Abstract

We study the average robustness notion in deep neural networks in (selected) wide and narrow, deep and shallow, as well as lazy and non-lazy training settings. We prove that in the under-parameterized setting, width has a negative effect while it improves robustness in the over-parameterized setting. The effect of depth closely depends on the initialization and the training mode. In particular, when initialized with LeCun initialization, depth helps robustness with the lazy training regime. In contrast, when initialized with Neural Tangent Kernel (NTK) and He-initialization, depth hurts the robustness. Moreover, under the non-lazy training regime, we demonstrate how the width of a two-layer ReLU network benefits robustness. Our theoretical developments improve the results by [Huang et al. NeurIPS21; Wu et al. NeurIPS21] and are consistent with [Bubeck and Sellke NeurIPS21; Bubeck et al. COLT21].