On Connections between Regularizations for Improving DNN Robustness

TL;DR

This paper provides a theoretical analysis of various regularization techniques aimed at enhancing the adversarial robustness of deep neural networks, revealing their connections and underlying principles to guide future improvements.

Contribution

It uncovers the relationships between input-gradient, Jacobian, curvature, and cross-Lipschitz regularizations for DNNs with ReLU activations, offering insights for designing better regularizations.

Findings

Identifies connections between different regularization methods.

Provides a theoretical reinterpretation of their functionalities.

Lays groundwork for developing more effective regularizations.

Abstract

This paper analyzes regularization terms proposed recently for improving the adversarial robustness of deep neural networks (DNNs), from a theoretical point of view. Specifically, we study possible connections between several effective methods, including input-gradient regularization, Jacobian regularization, curvature regularization, and a cross-Lipschitz functional. We investigate them on DNNs with general rectified linear activations, which constitute one of the most prevalent families of models for image classification and a host of other machine learning applications. We shed light on essential ingredients of these regularizations and re-interpret their functionality. Through the lens of our study, more principled and efficient regularizations can possibly be invented in the near future.