SynthesizRR: Generating Diverse Datasets with Retrieval Augmentation

TL;DR

SynthesizRR introduces retrieval-augmented dataset synthesis to enhance diversity and quality of generated data for training smaller models, outperforming previous prompting methods.

Contribution

The paper presents SynthesizRR, a novel retrieval-based approach that significantly improves dataset diversity and synthesis quality over traditional few-shot prompting methods.

Findings

Enhanced lexical and semantic diversity in synthesized datasets

Improved similarity to human-written text

Better distillation performance for smaller models

Abstract

It is often desirable to distill the capabilities of large language models (LLMs) into smaller student models due to compute and memory constraints. One way to do this for classification tasks is via dataset synthesis, which can be accomplished by generating examples of each label from the LLM. Prior approaches to synthesis use few-shot prompting, which relies on the LLM's parametric knowledge to generate usable examples. However, this leads to issues of repetition, bias towards popular entities, and stylistic differences from human text. In this work, we propose Synthesize by Retrieval and Refinement (SynthesizRR), which uses retrieval augmentation to introduce variety into the dataset synthesis process: as retrieved passages vary, the LLM is seeded with different content to generate its examples. We empirically study the synthesis of six datasets, covering topic classification, sentiment analysis, tone detection, and humor, requiring complex synthesis strategies. We find that SynthesizRR greatly improves lexical and semantic diversity, similarity to human-written text, and distillation performance, when compared to 32-shot prompting and four prior approaches. We release our code to perform all steps at https://github.com/amazon-science/synthesizrr