GradAug: A New Regularization Method for Deep Neural Networks

TL;DR

GradAug is a novel regularization technique for deep neural networks that uses random transformations to improve generalization, achieve state-of-the-art accuracy, and enhance robustness across various tasks and conditions.

Contribution

The paper introduces GradAug, a new gradient augmentation method that regularizes sub-networks via random transformations, improving performance and robustness in deep learning.

Findings

Achieves 78.79% top-1 accuracy on ImageNet with ResNet-50.

Further improves to 79.67% with CutMix, surpassing ensemble methods.

Enhances generalization in object detection, segmentation, and robustness to distortions and adversarial attacks.

Abstract

We propose a new regularization method to alleviate over-fitting in deep neural networks. The key idea is utilizing randomly transformed training samples to regularize a set of sub-networks, which are originated by sampling the width of the original network, in the training process. As such, the proposed method introduces self-guided disturbances to the raw gradients of the network and therefore is termed as Gradient Augmentation (GradAug). We demonstrate that GradAug can help the network learn well-generalized and more diverse representations. Moreover, it is easy to implement and can be applied to various structures and applications. GradAug improves ResNet-50 to 78.79% on ImageNet classification, which is a new state-of-the-art accuracy. By combining with CutMix, it further boosts the performance to 79.67%, which outperforms an ensemble of advanced training tricks. The generalization ability is evaluated on COCO object detection and instance segmentation where GradAug significantly surpasses other state-of-the-art methods. GradAug is also robust to image distortions and FGSM adversarial attacks and is highly effective in low data regimes. Code is available at https://github.com/taoyang1122/GradAug