Deep Augmentation: Dropout as Augmentation for Self-Supervised Learning

TL;DR

This paper investigates the use of dropout as a data augmentation technique in self-supervised learning, proposing Deep Augmentation which applies dropout or PCA at specific network layers to improve performance across various modalities.

Contribution

It introduces Deep Augmentation, a modality- and network-agnostic method applying dropout or PCA at targeted layers, with insights on when and how dropout enhances self-supervised learning.

Findings

Dropout is most effective in deeper layers for contrastive learning.

A stop-gradient operation is crucial for dropout's effectiveness as augmentation.

Deep Augmentation can outperform traditional input-level augmentations.

Abstract

Despite dropout's ubiquity in machine learning, its effectiveness as a form of data augmentation remains under-explored. We address two key questions: (i) When is dropout effective as an augmentation strategy? (ii) Is dropout uniquely effective under these conditions? To explore these questions, we propose Deep Augmentation, a network- and modality-agnostic method that applies dropout or PCA transformations to targeted layers in neural networks. Through extensive experiments on contrastive learning tasks in NLP, computer vision, and graph learning, we find that uniformly applying dropout across layers does not consistently improve performance. Instead, dropout proves most beneficial in deeper layers and can be matched by alternative augmentations (e.g., PCA). We also show that a stop-gradient operation is critical for ensuring dropout functions effectively as an augmentation, and that performance trends invert when moving from contrastive tasks to supervised tasks. Our analysis suggests that Deep Augmentation helps mitigate inter-layer co-adaptation -- a notable issue in self-supervised learning due to the absence of labeled data. Drawing on these insights, we outline a procedure for selecting the optimal augmentation layer and demonstrate that Deep Augmentation can outperform traditional input-level augmentations. This simple yet powerful approach can be seamlessly integrated into a wide range of architectures and modalities, yielding notable gains in both performance and generalization.