Rethinking Softmax Cross-Entropy Loss for Adversarial Robustness

TL;DR

This paper introduces the MMC loss, a novel training objective that promotes dense feature regions to enhance adversarial robustness without sacrificing standard accuracy.

Contribution

The paper identifies limitations of softmax cross-entropy loss for robustness and proposes MMC loss to explicitly induce dense, well-structured feature representations.

Findings

MMC loss improves adversarial robustness under strong attacks

Maintains state-of-the-art accuracy on clean data

Requires little extra computation compared to SCE loss

Abstract

Previous work shows that adversarially robust generalization requires larger sample complexity, and the same dataset, e.g., CIFAR-10, which enables good standard accuracy may not suffice to train robust models. Since collecting new training data could be costly, we focus on better utilizing the given data by inducing the regions with high sample density in the feature space, which could lead to locally sufficient samples for robust learning. We first formally show that the softmax cross-entropy (SCE) loss and its variants convey inappropriate supervisory signals, which encourage the learned feature points to spread over the space sparsely in training. This inspires us to propose the Max-Mahalanobis center (MMC) loss to explicitly induce dense feature regions in order to benefit robustness. Namely, the MMC loss encourages the model to concentrate on learning ordered and compact representations, which gather around the preset optimal centers for different classes. We empirically demonstrate that applying the MMC loss can significantly improve robustness even under strong adaptive attacks, while keeping state-of-the-art accuracy on clean inputs with little extra computation compared to the SCE loss.