S2O: Enhancing Adversarial Training with Second-Order Statistics of Weights

TL;DR

This paper introduces S2O, a novel adversarial training method that leverages second-order statistics of model weights to improve neural network robustness and generalization, supported by theoretical bounds and extensive experiments.

Contribution

It proposes a new approach that relaxes independence assumptions in PAC-Bayesian frameworks and optimizes second-order weight statistics to enhance adversarial training.

Findings

S2O improves robustness and generalization of neural networks.

S2O tightens PAC-Bayesian robust generalization bounds.

S2O complements existing adversarial training methods effectively.

Abstract

Adversarial training has emerged as a highly effective way to improve the robustness of deep neural networks (DNNs). It is typically conceptualized as a min-max optimization problem over model weights and adversarial perturbations, where the weights are optimized using gradient descent methods, such as SGD. In this paper, we propose a novel approach by treating model weights as random variables, which paves the way for enhancing adversarial training through \textbf{S}econd-Order \textbf{S}tatistics \textbf{O}ptimization (S$^2$O) over model weights. We challenge and relax a prevalent, yet often unrealistic, assumption in prior PAC-Bayesian frameworks: the statistical independence of weights. From this relaxation, we derive an improved PAC-Bayesian robust generalization bound. Our theoretical developments suggest that optimizing the second-order statistics of weights can substantially tighten this bound. We complement this theoretical insight by conducting an extensive set of experiments that demonstrate that S$^2$O not only enhances the robustness and generalization of neural networks when used in isolation, but also seamlessly augments other state-of-the-art adversarial training techniques. The code is available at https://github.com/Alexkael/S2O.