Unsupervised Data Augmentation for Consistency Training

TL;DR

This paper introduces a semi-supervised learning method that leverages advanced data augmentation techniques for consistency training, significantly improving performance across various language and vision tasks with limited labeled data.

Contribution

The paper demonstrates that using sophisticated data augmentation methods in consistency training enhances semi-supervised learning effectiveness across multiple domains.

Findings

Achieves 4.20% error on IMDb with only 20 labeled examples.

Outperforms previous methods on CIFAR-10 with only 250 labeled examples.

Improves ImageNet results with limited labeled data.

Abstract

Semi-supervised learning lately has shown much promise in improving deep learning models when labeled data is scarce. Common among recent approaches is the use of consistency training on a large amount of unlabeled data to constrain model predictions to be invariant to input noise. In this work, we present a new perspective on how to effectively noise unlabeled examples and argue that the quality of noising, specifically those produced by advanced data augmentation methods, plays a crucial role in semi-supervised learning. By substituting simple noising operations with advanced data augmentation methods such as RandAugment and back-translation, our method brings substantial improvements across six language and three vision tasks under the same consistency training framework. On the IMDb text classification dataset, with only 20 labeled examples, our method achieves an error rate of 4.20, outperforming the state-of-the-art model trained on 25,000 labeled examples. On a standard semi-supervised learning benchmark, CIFAR-10, our method outperforms all previous approaches and achieves an error rate of 5.43 with only 250 examples. Our method also combines well with transfer learning, e.g., when finetuning from BERT, and yields improvements in high-data regime, such as ImageNet, whether when there is only 10% labeled data or when a full labeled set with 1.3M extra unlabeled examples is used. Code is available at https://github.com/google-research/uda.