Bridging the Gap: Rademacher Complexity in Robust and Standard Generalization

TL;DR

This paper develops new theoretical bounds on the Rademacher complexity for adversarially trained deep neural networks, matching standard bounds and addressing the challenge of adversarial function class covering.

Contribution

It introduces a novel uniform covering number that enables deriving tight Rademacher complexity bounds for adversarial DNNs, bridging the gap with standard generalization bounds.

Findings

Achieves upper bounds with $ ext{O}( ext{ln}(dm))$ dependency on width and dimension.

Introduces the uniform covering number for adversarial function classes.

Bridges the theoretical gap between robust and standard generalization bounds.

Abstract

Training Deep Neural Networks (DNNs) with adversarial examples often results in poor generalization to test-time adversarial data. This paper investigates this issue, known as adversarially robust generalization, through the lens of Rademacher complexity. Building upon the studies by Khim and Loh (2018); Yin et al. (2019), numerous works have been dedicated to this problem, yet achieving a satisfactory bound remains an elusive goal. Existing works on DNNs either apply to a surrogate loss instead of the robust loss or yield bounds that are notably looser compared to their standard counterparts. In the latter case, the bounds have a higher dependency on the width $m$ of the DNNs or the dimension $d$ of the data, with an extra factor of at least $\mathcal{O}(\sqrt{m})$ or $\mathcal{O}(\sqrt{d})$. This paper presents upper bounds for adversarial Rademacher complexity of DNNs that match the best-known upper bounds in standard settings, as established in the work of Bartlett et al. (2017), with the dependency on width and dimension being $\mathcal{O}(\ln(dm))$. The central challenge addressed is calculating the covering number of adversarial function classes. We aim to construct a new cover that possesses two properties: 1) compatibility with adversarial examples, and 2) precision comparable to covers used in standard settings. To this end, we introduce a new variant of covering number called the \emph{uniform covering number}, specifically designed and proven to reconcile these two properties. Consequently, our method effectively bridges the gap between Rademacher complexity in robust and standard generalization.