Self-training Improves Pre-training for Natural Language Understanding

TL;DR

This paper demonstrates that self-training combined with a novel data augmentation method, SentAugment, enhances pre-training for natural language understanding, leading to significant improvements across multiple tasks without requiring in-domain unlabeled data.

Contribution

The paper introduces SentAugment, a task-specific data augmentation technique that enables scalable self-training using web-crawled unlabeled sentences, improving NLP performance without in-domain data.

Findings

Up to 2.6% improvement on text classification benchmarks

Effective in knowledge-distillation and few-shot learning scenarios

Self-training complements strong pre-trained models like RoBERTa

Abstract

Unsupervised pre-training has led to much recent progress in natural language understanding. In this paper, we study self-training as another way to leverage unlabeled data through semi-supervised learning. To obtain additional data for a specific task, we introduce SentAugment, a data augmentation method which computes task-specific query embeddings from labeled data to retrieve sentences from a bank of billions of unlabeled sentences crawled from the web. Unlike previous semi-supervised methods, our approach does not require in-domain unlabeled data and is therefore more generally applicable. Experiments show that self-training is complementary to strong RoBERTa baselines on a variety of tasks. Our augmentation approach leads to scalable and effective self-training with improvements of up to 2.6% on standard text classification benchmarks. Finally, we also show strong gains on knowledge-distillation and few-shot learning.