Improving the Robustness and Generalization of Deep Neural Network with Confidence Threshold Reduction

TL;DR

This paper introduces confidence threshold reduction (CTR) techniques, including MDL and STD loss functions, to enhance the robustness and generalization of deep neural networks against adversarial attacks and natural training limitations.

Contribution

It proposes novel loss functions for CTR that improve both robustness and generalization in natural and adversarial training scenarios.

Findings

MDL improves robustness and generalization in natural training.

STD loss further enhances adversarial training robustness.

Theoretical analysis supports the effectiveness of CTR methods.

Abstract

Deep neural networks are easily attacked by imperceptible perturbation. Presently, adversarial training (AT) is the most effective method to enhance the robustness of the model against adversarial examples. However, because adversarial training solved a min-max value problem, in comparison with natural training, the robustness and generalization are contradictory, i.e., the robustness improvement of the model will decrease the generalization of the model. To address this issue, in this paper, a new concept, namely confidence threshold (CT), is introduced and the reducing of the confidence threshold, known as confidence threshold reduction (CTR), is proven to improve both the generalization and robustness of the model. Specifically, to reduce the CT for natural training (i.e., for natural training with CTR), we propose a mask-guided divergence loss function (MDL) consisting of a cross-entropy loss term and an orthogonal term. The empirical and theoretical analysis demonstrates that the MDL loss improves the robustness and generalization of the model simultaneously for natural training. However, the model robustness improvement of natural training with CTR is not comparable to that of adversarial training. Therefore, for adversarial training, we propose a standard deviation loss function (STD), which minimizes the difference in the probabilities of the wrong categories, to reduce the CT by being integrated into the loss function of adversarial training. The empirical and theoretical analysis demonstrates that the STD based loss function can further improve the robustness of the adversarially trained model on basis of guaranteeing the changeless or slight improvement of the natural accuracy.