Characterizing Model Robustness via Natural Input Gradients

TL;DR

This paper demonstrates that regularizing the input gradients on natural examples, especially in vision transformers with smooth activations, can achieve robustness comparable to adversarial training with less computational cost.

Contribution

It reveals the effectiveness of input gradient regularization on natural data, especially for smooth activation models, challenging prior beliefs about its limitations.

Findings

Gradient Norm regularization performs well on vision transformers with smooth activations.

Achieves over 90% of adversarial training accuracy on ImageNet-1k.

Regularizing gradients on image edges improves robustness without explicit gradient norm constraints.

Abstract

Adversarially robust models are locally smooth around each data sample so that small perturbations cannot drastically change model outputs. In modern systems, such smoothness is usually obtained via Adversarial Training, which explicitly enforces models to perform well on perturbed examples. In this work, we show the surprising effectiveness of instead regularizing the gradient with respect to model inputs on natural examples only. Penalizing input Gradient Norm is commonly believed to be a much inferior approach. Our analyses identify that the performance of Gradient Norm regularization critically depends on the smoothness of activation functions, and are in fact extremely effective on modern vision transformers that adopt smooth activations over piecewise linear ones (eg, ReLU), contrary to prior belief. On ImageNet-1k, Gradient Norm training achieves > 90% the performance of state-of-the-art PGD-3 Adversarial Training} (52% vs.~56%), while using only 60% computation cost of the state-of-the-art without complex adversarial optimization. Our analyses also highlight the relationship between model robustness and properties of natural input gradients, such as asymmetric sample and channel statistics. Surprisingly, we find model robustness can be significantly improved by simply regularizing its gradients to concentrate on image edges without explicit conditioning on the gradient norm.