Lower Difficulty and Better Robustness: A Bregman Divergence Perspective for Adversarial Training

TL;DR

This paper introduces a Bregman divergence perspective to analyze adversarial training, revealing that easier optimization processes and higher entropy models lead to better robustness, and proposes methods to improve training difficulty and robustness.

Contribution

The paper presents a novel Bregman divergence framework for adversarial training and proposes two methods, FAIT and MER, to reduce optimization difficulty and enhance robustness.

Findings

TRADES separates PGD-AT, making it easier to optimize.

High entropy models exhibit better robustness.

Proposed methods improve robustness under 10-step PGD adversaries.

Abstract

In this paper, we investigate on improving the adversarial robustness obtained in adversarial training (AT) via reducing the difficulty of optimization. To better study this problem, we build a novel Bregman divergence perspective for AT, in which AT can be viewed as the sliding process of the training data points on the negative entropy curve. Based on this perspective, we analyze the learning objectives of two typical AT methods, i.e., PGD-AT and TRADES, and we find that the optimization process of TRADES is easier than PGD-AT for that TRADES separates PGD-AT. In addition, we discuss the function of entropy in TRADES, and we find that models with high entropy can be better robustness learners. Inspired by the above findings, we propose two methods, i.e., FAIT and MER, which can both not only reduce the difficulty of optimization under the 10-step PGD adversaries, but also provide better robustness. Our work suggests that reducing the difficulty of optimization under the 10-step PGD adversaries is a promising approach for enhancing the adversarial robustness in AT.