Adversarial Training for Gradient Descent: Analysis Through its Continuous-time Approximation

TL;DR

This paper introduces a continuous-time approximation for adversarial training, providing analytical insights into its robustness and differences from standard gradient descent, supported by theoretical and numerical evidence.

Contribution

It offers a novel continuous-time framework to analyze adversarial training, enhancing understanding of its robustness compared to traditional gradient descent methods.

Findings

Adversarial training can be approximated by a continuous-time gradient flow.

The analysis confirms the robustness of adversarial training.

Numerical examples support the theoretical findings.

Abstract

Adversarial training has gained great popularity as one of the most effective defenses for deep neural network and more generally for gradient-based machine learning models against adversarial perturbations on data points. This paper establishes a continuous-time approximation for the mini-max game of adversarial training. This approximation approach allows for precise and analytical comparisons between stochastic gradient descent and its adversarial training counterpart; and confirms theoretically the robustness of adversarial training from a new gradient-flow viewpoint. The analysis is then corroborated through various analytical and numerical examples.