Is linguistically-motivated data augmentation worth it?

TL;DR

This paper systematically compares linguistically-naive and linguistically-motivated data augmentation methods for low-resource languages, finding that linguistically-motivated strategies can improve performance if the synthetic data closely resembles the original data distribution.

Contribution

It provides the first comprehensive empirical evaluation of both augmentation strategies across two low-resource languages and two sequence-to-sequence tasks.

Findings

Linguistically-motivated augmentation can outperform naive methods.

Effectiveness depends on the similarity of synthetic data to training data.

Strategies show varied benefits across languages and tasks.

Abstract

Data augmentation, a widely-employed technique for addressing data scarcity, involves generating synthetic data examples which are then used to augment available training data. Researchers have seen surprising success from simple methods, such as random perturbations from natural examples, where models seem to benefit even from data with nonsense words, or data that doesn't conform to the rules of the language. A second line of research produces synthetic data that does in fact follow all linguistic constraints; these methods require some linguistic expertise and are generally more challenging to implement. No previous work has done a systematic, empirical comparison of both linguistically-naive and linguistically-motivated data augmentation strategies, leaving uncertainty about whether the additional time and effort of linguistically-motivated data augmentation work in fact yields better downstream performance. In this work, we conduct a careful and comprehensive comparison of augmentation strategies (both linguistically-naive and linguistically-motivated) for two low-resource languages with different morphological properties, Uspanteko and Arapaho. We evaluate the effectiveness of many different strategies and their combinations across two important sequence-to-sequence tasks for low-resource languages: machine translation and interlinear glossing. We find that linguistically-motivated strategies can have benefits over naive approaches, but only when the new examples they produce are not significantly unlike the training data distribution.