Text Data Augmentation for Large Language Models: A Comprehensive Survey of Methods, Challenges, and Opportunities

TL;DR

This survey reviews various data augmentation techniques for large language models, highlighting methods, challenges, and future directions to enhance model training with limited data.

Contribution

It provides a comprehensive classification and analysis of data augmentation methods for LLMs, including recent retrieval-based and hybrid approaches.

Findings

Retrieval-based techniques improve data grounding.

Post-processing refines augmented data.

Hybrid methods combine multiple augmentation strategies.

Abstract

The increasing size and complexity of pre-trained language models have demonstrated superior performance in many applications, but they usually require large training datasets to be adequately trained. Insufficient training sets could unexpectedly make the model overfit and fail to cope with complex tasks. Large language models (LLMs) trained on extensive corpora have prominent text generation capabilities, which improve the quality and quantity of data and play a crucial role in data augmentation. Specifically, distinctive prompt templates are given in personalised tasks to guide LLMs in generating the required content. Recent promising retrieval-based techniques further improve the expressive performance of LLMs in data augmentation by introducing external knowledge to enable them to produce more grounded-truth data. This survey provides an in-depth analysis of data augmentation in LLMs, classifying the techniques into Simple Augmentation, Prompt-based Augmentation, Retrieval-based Augmentation and Hybrid Augmentation. We summarise the post-processing approaches in data augmentation, which contributes significantly to refining the augmented data and enabling the model to filter out unfaithful content. Then, we provide the common tasks and evaluation metrics. Finally, we introduce existing challenges and future opportunities that could bring further improvement to data augmentation.