Better Synthetic Data by Retrieving and Transforming Existing Datasets

TL;DR

DataTune leverages existing datasets by transforming and repurposing them to generate more diverse and complex training data, significantly enhancing NLP model performance over traditional prompt-based synthetic data methods.

Contribution

Introduces DataTune, a novel dataset transformation approach that improves synthetic data quality and diversity by utilizing existing datasets for specific NLP tasks.

Findings

DataTune improves model performance by 49% over few-shot prompting.

It outperforms existing synthetic data methods by 34%.

Dataset transformation increases data diversity and difficulty.

Abstract

Despite recent advances in large language models, building dependable and deployable NLP models typically requires abundant, high-quality training data. However, task-specific data is not available for many use cases, and manually curating task-specific data is labor-intensive. Recent work has studied prompt-driven synthetic data generation using large language models, but these generated datasets tend to lack complexity and diversity. To address these limitations, we introduce a method, DataTune, to make better use of existing, publicly available datasets to improve automatic dataset generation. DataTune performs dataset transformation, enabling the repurposing of publicly available datasets into a format that is directly aligned with the specific requirements of target tasks. On a diverse set of language-based tasks from the BIG-Bench benchmark, we find that finetuning language models via DataTune improves over a few-shot prompting baseline by 49% and improves over existing methods that use synthetic or retrieved training data by 34%. We find that dataset transformation significantly increases the diversity and difficulty of generated data on many tasks. We integrate DataTune into an open-source repository to make this method accessible to the community: https://github.com/neulab/prompt2model.