Low Curvature Activations Reduce Overfitting in Adversarial Training

TL;DR

This paper demonstrates that using activation functions with low curvature can significantly reduce overfitting and the generalization gap in adversarial training, improving robustness and preventing double descent.

Contribution

It reveals the impact of low-curvature activation functions on reducing overfitting and generalization gaps in adversarial training, including non-smooth activations like LeakyReLU.

Findings

Low-curvature activations reduce standard and robust generalization gaps.

Activation functions with low curvature prevent the double descent phenomenon.

Both smooth and non-smooth low-curvature activations have regularization effects.

Abstract

Adversarial training is one of the most effective defenses against adversarial attacks. Previous works suggest that overfitting is a dominant phenomenon in adversarial training leading to a large generalization gap between test and train accuracy in neural networks. In this work, we show that the observed generalization gap is closely related to the choice of the activation function. In particular, we show that using activation functions with low (exact or approximate) curvature values has a regularization effect that significantly reduces both the standard and robust generalization gaps in adversarial training. We observe this effect for both differentiable/smooth activations such as SiLU as well as non-differentiable/non-smooth activations such as LeakyReLU. In the latter case, the "approximate" curvature of the activation is low. Finally, we show that for activation functions with low curvature, the double descent phenomenon for adversarially trained models does not occur.