Average Margin Regularization for Classifiers

TL;DR

This paper introduces Average Margin (AM) regularization, a novel technique that improves both accuracy and adversarial robustness of classifiers by balancing margin maximization, supported by theoretical analysis and empirical results.

Contribution

It proposes a new AM regularization method that enhances classifier accuracy and robustness, challenging the existing tradeoff suggested by prior theoretical results.

Findings

AM regularization improves SVM accuracy and robustness

Theoretical bounds support the benefits of AM regularization

Empirical results on synthetic and real data validate the approach

Abstract

Adversarial robustness has become an important research topic given empirical demonstrations on the lack of robustness of deep neural networks. Unfortunately, recent theoretical results suggest that adversarial training induces a strict tradeoff between classification accuracy and adversarial robustness. In this paper, we propose and then study a new regularization for any margin classifier or deep neural network. We motivate this regularization by a novel generalization bound that shows a tradeoff in classifier accuracy between maximizing its margin and average margin. We thus call our approach an average margin (AM) regularization, and it consists of a linear term added to the objective. We theoretically show that for certain distributions AM regularization can both improve classifier accuracy and robustness to adversarial attacks. We conclude by using both synthetic and real data to empirically show that AM regularization can strictly improve both accuracy and robustness for support vector machine's (SVM's), relative to unregularized classifiers and adversarially trained classifiers.