Label Smoothing and Logit Squeezing: A Replacement for Adversarial Training?

TL;DR

This paper investigates how simple regularization techniques like label smoothing and logit squeezing, combined with Gaussian noise, can replicate and even surpass the robustness benefits of traditional adversarial training without needing adversarial examples.

Contribution

The study demonstrates that regularization methods can effectively mimic adversarial training mechanisms, providing a more efficient way to enhance classifier robustness.

Findings

Regularization methods can match or outperform adversarial training in robustness.

Combining label smoothing, logit squeezing, and Gaussian noise achieves strong adversarial robustness.

No adversarial examples are needed to attain high robustness.

Abstract

Adversarial training is one of the strongest defenses against adversarial attacks, but it requires adversarial examples to be generated for every mini-batch during optimization. The expense of producing these examples during training often precludes adversarial training from use on complex image datasets. In this study, we explore the mechanisms by which adversarial training improves classifier robustness, and show that these mechanisms can be effectively mimicked using simple regularization methods, including label smoothing and logit squeezing. Remarkably, using these simple regularization methods in combination with Gaussian noise injection, we are able to achieve strong adversarial robustness -- often exceeding that of adversarial training -- using no adversarial examples.