SSMBA: Self-Supervised Manifold Based Data Augmentation for Improving Out-of-Domain Robustness

TL;DR

SSMBA introduces a self-supervised data augmentation technique that enhances out-of-domain robustness in NLP models by generating synthetic examples through manifold-based corruption and reconstruction.

Contribution

It proposes a novel manifold-based augmentation method using corruption and reconstruction, improving out-of-domain generalization in NLP tasks.

Findings

Outperforms existing augmentation methods on robustness benchmarks

Achieves 0.8% accuracy gain on OOD Amazon reviews

Achieves 1.8% accuracy gain on OOD MNLI

Abstract

Models that perform well on a training domain often fail to generalize to out-of-domain (OOD) examples. Data augmentation is a common method used to prevent overfitting and improve OOD generalization. However, in natural language, it is difficult to generate new examples that stay on the underlying data manifold. We introduce SSMBA, a data augmentation method for generating synthetic training examples by using a pair of corruption and reconstruction functions to move randomly on a data manifold. We investigate the use of SSMBA in the natural language domain, leveraging the manifold assumption to reconstruct corrupted text with masked language models. In experiments on robustness benchmarks across 3 tasks and 9 datasets, SSMBA consistently outperforms existing data augmentation methods and baseline models on both in-domain and OOD data, achieving gains of 0.8% accuracy on OOD Amazon reviews, 1.8% accuracy on OOD MNLI, and 1.4 BLEU on in-domain IWSLT14 German-English.