PAC-Bayesian Spectrally-Normalized Bounds for Adversarially Robust Generalization

TL;DR

This paper derives a spectrally-normalized PAC-Bayesian bound for adversarially robust generalization in deep neural networks, addressing previous limitations and extending to various attack types and architectures.

Contribution

It introduces a tighter, assumption-free robust generalization bound based on spectral normalization, offering new insights into the disparity between standard and robust generalization.

Findings

The bound is tighter and assumption-free compared to previous work.

It applies to general non-$ ext{ell}_p$ attacks and diverse neural network architectures.

Disparities in standard and robust bounds are due to mathematical issues, not fundamental limitations.

Abstract

Deep neural networks (DNNs) are vulnerable to adversarial attacks. It is found empirically that adversarially robust generalization is crucial in establishing defense algorithms against adversarial attacks. Therefore, it is interesting to study the theoretical guarantee of robust generalization. This paper focuses on norm-based complexity, based on a PAC-Bayes approach (Neyshabur et al., 2017). The main challenge lies in extending the key ingredient, which is a weight perturbation bound in standard settings, to the robust settings. Existing attempts heavily rely on additional strong assumptions, leading to loose bounds. In this paper, we address this issue and provide a spectrally-normalized robust generalization bound for DNNs. Compared to existing bounds, our bound offers two significant advantages: Firstly, it does not depend on additional assumptions. Secondly, it is considerably tighter, aligning with the bounds of standard generalization. Therefore, our result provides a different perspective on understanding robust generalization: The mismatch terms between standard and robust generalization bounds shown in previous studies do not contribute to the poor robust generalization. Instead, these disparities solely due to mathematical issues. Finally, we extend the main result to adversarial robustness against general non-$\ell_p$ attacks and other neural network architectures.