Data augmentation instead of explicit regularization

TL;DR

This paper demonstrates that data augmentation alone can match or surpass traditional regularization techniques like weight decay and dropout in deep learning, simplifying training and reducing hyperparameter tuning.

Contribution

It provides formal definitions of explicit and implicit regularization, and empirically shows data augmentation's effectiveness over weight decay and dropout in image classification tasks.

Findings

Data augmentation achieves comparable or better performance than weight decay and dropout.

Hyperparameter tuning of weight decay and dropout can harm performance if not carefully done.

Data augmentation provides significant generalization gains without additional hyperparameter tuning.

Abstract

Contrary to most machine learning models, modern deep artificial neural networks typically include multiple components that contribute to regularization. Despite the fact that some (explicit) regularization techniques, such as weight decay and dropout, require costly fine-tuning of sensitive hyperparameters, the interplay between them and other elements that provide implicit regularization is not well understood yet. Shedding light upon these interactions is key to efficiently using computational resources and may contribute to solving the puzzle of generalization in deep learning. Here, we first provide formal definitions of explicit and implicit regularization that help understand essential differences between techniques. Second, we contrast data augmentation with weight decay and dropout. Our results show that visual object categorization models trained with data augmentation alone achieve the same performance or higher than models trained also with weight decay and dropout, as is common practice. We conclude that the contribution on generalization of weight decay and dropout is not only superfluous when sufficient implicit regularization is provided, but also such techniques can dramatically deteriorate the performance if the hyperparameters are not carefully tuned for the architecture and data set. In contrast, data augmentation systematically provides large generalization gains and does not require hyperparameter re-tuning. In view of our results, we suggest to optimize neural networks without weight decay and dropout to save computational resources, hence carbon emissions, and focus more on data augmentation and other inductive biases to improve performance and robustness.