GenAug: Data Augmentation for Finetuning Text Generators

TL;DR

This paper explores data augmentation techniques for improving text generation quality, especially in low-data scenarios, by evaluating methods like synthetic noise and hypernym replacement on Yelp reviews.

Contribution

It introduces and assesses various augmentation methods for finetuning GPT-2, highlighting effective techniques and optimal data augmentation levels for better text generation.

Findings

Insertion of character-level synthetic noise improves diversity.

Keyword replacement with hypernyms enhances fluency.

Optimal augmentation is around three times the original data.

Abstract

In this paper, we investigate data augmentation for text generation, which we call GenAug. Text generation and language modeling are important tasks within natural language processing, and are especially challenging for low-data regimes. We propose and evaluate various augmentation methods, including some that incorporate external knowledge, for finetuning GPT-2 on a subset of Yelp Reviews. We also examine the relationship between the amount of augmentation and the quality of the generated text. We utilize several metrics that evaluate important aspects of the generated text including its diversity and fluency. Our experiments demonstrate that insertion of character-level synthetic noise and keyword replacement with hypernyms are effective augmentation methods, and that the quality of generations improves to a peak at approximately three times the amount of original data.