Distribution-restrained Softmax Loss for the Model Robustness

TL;DR

This paper introduces a novel loss function that reduces the diversity of softmax output distributions for non-real labels, thereby enhancing the robustness of deep learning models against attacks efficiently.

Contribution

The paper proposes a distribution-restrained softmax loss that improves model robustness by controlling softmax distribution characteristics, a principle not fully explored before.

Findings

Improves robustness without significant computational overhead

Softmax distribution characteristics are highly correlated with attack results

Method enhances model resilience against adversarial attacks

Abstract

Recently, the robustness of deep learning models has received widespread attention, and various methods for improving model robustness have been proposed, including adversarial training, model architecture modification, design of loss functions, certified defenses, and so on. However, the principle of the robustness to attacks is still not fully understood, also the related research is still not sufficient. Here, we have identified a significant factor that affects the robustness of models: the distribution characteristics of softmax values for non-real label samples. We found that the results after an attack are highly correlated with the distribution characteristics, and thus we proposed a loss function to suppress the distribution diversity of softmax. A large number of experiments have shown that our method can improve robustness without significant time consumption.