Adversarial Vulnerability as a Consequence of On-Manifold Inseparibility

TL;DR

This paper links adversarial vulnerability to the difficulty of training off-manifold features due to ill-conditioning, proposing second-order optimization methods to improve robustness and analyzing the impact of batch normalization.

Contribution

It introduces a theoretical framework connecting data manifold properties with adversarial vulnerability and demonstrates the effectiveness of second-order methods in enhancing robustness.

Findings

Second-order methods improve robustness in training.

Batch normalization hinders robustness gains.

Ill-conditioning affects off-manifold feature convergence.

Abstract

Recent works have shown theoretically and empirically that redundant data dimensions are a source of adversarial vulnerability. However, the inverse doesn't seem to hold in practice; employing dimension-reduction techniques doesn't exhibit robustness as expected. In this work, we consider classification tasks and characterize the data distribution as a low-dimensional manifold, with high/low variance features defining the on/off manifold direction. We argue that clean training experiences poor convergence in the off-manifold direction caused by the ill-conditioning in widely used first-order optimizers like gradient descent. The poor convergence then acts as a source of adversarial vulnerability when the dataset is inseparable in the on-manifold direction. We provide theoretical results for logistic regression and a 2-layer linear network on the considered data distribution. Furthermore, we advocate using second-order methods that are immune to ill-conditioning and lead to better robustness. We perform experiments and exhibit tremendous robustness improvements in clean training through long training and the employment of second-order methods, corroborating our framework. Additionally, we find the inclusion of batch-norm layers hinders such robustness gains. We attribute this to differing implicit biases between traditional and batch-normalized neural networks.