MMA Training: Direct Input Space Margin Maximization through Adversarial Training

TL;DR

This paper introduces MMA training, a novel method that directly maximizes input space margins to improve neural network robustness against adversarial attacks, with adaptive margin selection per data point.

Contribution

It proposes MMA training, a new adversarial training approach that maximizes margins directly and adaptively, providing a theoretical analysis and empirical validation on standard datasets.

Findings

MMA training improves robustness on MNIST and CIFAR10.

Theoretical connection between adversarial loss and margin maximization.

Adaptive margin selection enhances robustness over fixed epsilon methods.

Abstract

We study adversarial robustness of neural networks from a margin maximization perspective, where margins are defined as the distances from inputs to a classifier's decision boundary. Our study shows that maximizing margins can be achieved by minimizing the adversarial loss on the decision boundary at the "shortest successful perturbation", demonstrating a close connection between adversarial losses and the margins. We propose Max-Margin Adversarial (MMA) training to directly maximize the margins to achieve adversarial robustness. Instead of adversarial training with a fixed $\epsilon$, MMA offers an improvement by enabling adaptive selection of the "correct" $\epsilon$ as the margin individually for each datapoint. In addition, we rigorously analyze adversarial training with the perspective of margin maximization, and provide an alternative interpretation for adversarial training, maximizing either a lower or an upper bound of the margins. Our experiments empirically confirm our theory and demonstrate MMA training's efficacy on the MNIST and CIFAR10 datasets w.r.t. $\ell_\infty$ and $\ell_2$ robustness. Code and models are available at https://github.com/BorealisAI/mma_training.