Stationary Point Losses for Robust Model

TL;DR

This paper introduces stationary point (SP) losses to improve neural network robustness by ensuring decision boundaries are more centered between classes, thus resisting adversarial attacks better.

Contribution

The paper proposes a novel family of SP losses that guarantee robust decision boundaries and improve adversarial robustness without significant accuracy loss.

Findings

SP loss enhances robustness against various adversarial attacks

Robust boundaries learned by SP loss perform well on imbalanced datasets

SP loss requires larger perturbations for adversarial example generation

Abstract

The inability to guarantee robustness is one of the major obstacles to the application of deep learning models in security-demanding domains. We identify that the most commonly used cross-entropy (CE) loss does not guarantee robust boundary for neural networks. CE loss sharpens the neural network at the decision boundary to achieve a lower loss, rather than pushing the boundary to a more robust position. A robust boundary should be kept in the middle of samples from different classes, thus maximizing the margins from the boundary to the samples. We think this is due to the fact that CE loss has no stationary point. In this paper, we propose a family of new losses, called stationary point (SP) loss, which has at least one stationary point on the correct classification side. We proved that robust boundary can be guaranteed by SP loss without losing much accuracy. With SP loss, larger perturbations are required to generate adversarial examples. We demonstrate that robustness is improved under a variety of adversarial attacks by applying SP loss. Moreover, robust boundary learned by SP loss also performs well on imbalanced datasets.