Gradient Regularization Improves Accuracy of Discriminative Models

TL;DR

This paper shows that gradient regularization enhances the accuracy of discriminative neural models, especially with limited data, by controlling Jacobian norms to improve generalization in vision tasks.

Contribution

It introduces a class of Jacobian-based gradient regularizers and provides empirical evidence of their effectiveness in improving neural network accuracy and generalization.

Findings

Gradient regularization improves vision model accuracy.

Regularizers control gradients beyond training points.

Enhanced generalization with small datasets.

Abstract

Regularizing the gradient norm of the output of a neural network with respect to its inputs is a powerful technique, rediscovered several times. This paper presents evidence that gradient regularization can consistently improve classification accuracy on vision tasks, using modern deep neural networks, especially when the amount of training data is small. We introduce our regularizers as members of a broader class of Jacobian-based regularizers. We demonstrate empirically on real and synthetic data that the learning process leads to gradients controlled beyond the training points, and results in solutions that generalize well.