Semi-Implicit Hybrid Gradient Methods with Application to Adversarial Robustness

TL;DR

This paper introduces semi-implicit hybrid gradient methods for adversarial training of neural networks, achieving faster convergence and improved robustness over existing algorithms by solving nonconvex-nonconcave minimax problems.

Contribution

It generalizes the stochastic primal-dual hybrid gradient algorithm to develop SI-HGs with $O(1/K)$ convergence rate for adversarial robustness training.

Findings

SI-HGs outperform existing AT algorithms in convergence speed.

SI-HGs demonstrate enhanced robustness in adversarial training.

Practical variants of SI-HGs are effective in real-world scenarios.

Abstract

Adversarial examples, crafted by adding imperceptible perturbations to natural inputs, can easily fool deep neural networks (DNNs). One of the most successful methods for training adversarially robust DNNs is solving a nonconvex-nonconcave minimax problem with an adversarial training (AT) algorithm. However, among the many AT algorithms, only Dynamic AT (DAT) and You Only Propagate Once (YOPO) guarantee convergence to a stationary point. In this work, we generalize the stochastic primal-dual hybrid gradient algorithm to develop semi-implicit hybrid gradient methods (SI-HGs) for finding stationary points of nonconvex-nonconcave minimax problems. SI-HGs have the convergence rate $O(1/K)$, which improves upon the rate $O(1/K^{1/2})$ of DAT and YOPO. We devise a practical variant of SI-HGs, and show that it outperforms other AT algorithms in terms of convergence speed and robustness.