Exact Spectral Norm Regularization for Neural Networks

TL;DR

This paper introduces an exact spectral norm regularization method for neural networks that improves generalization and robustness against noise, challenging previous assumptions about Jacobian-based adversarial defenses.

Contribution

We develop a scheme that directly targets the exact spectral norm of the Jacobian, surpassing prior upper bound methods in effectiveness.

Findings

Enhanced generalization performance over previous spectral regularization methods

Strong robustness against natural and adversarial noise

Reevaluation of Jacobian regularization's role in adversarial protection

Abstract

We pursue a line of research that seeks to regularize the spectral norm of the Jacobian of the input-output mapping for deep neural networks. While previous work rely on upper bounding techniques, we provide a scheme that targets the exact spectral norm. We showcase that our algorithm achieves an improved generalization performance compared to previous spectral regularization techniques while simultaneously maintaining a strong safeguard against natural and adversarial noise. Moreover, we further explore some previous reasoning concerning the strong adversarial protection that Jacobian regularization provides and show that it can be misleading.