Synthetic Data Generation in Low-Resource Settings via Fine-Tuning of Large Language Models

TL;DR

This paper explores using fine-tuned large language models to generate synthetic training data, significantly enhancing the performance of smaller models in text classification and generation tasks with minimal original data.

Contribution

It introduces a method of synthetic data generation via fine-tuned teacher LLMs to boost small model performance, reducing the need for extensive labeled datasets.

Findings

Synthetic data improves downstream task performance.

Small models achieve comparable results with less data.

Synthetic data generation reduces labeling costs.

Abstract

The in-context learning ability of large language models (LLMs) enables them to generalize to novel downstream tasks with relatively few labeled examples. However, they require enormous computational resources to be deployed. Alternatively, smaller models can solve specific tasks if fine-tuned with enough labeled examples. These examples, however, are expensive to obtain. In pursuit of the best of both worlds, we study synthetic data generation of fine-tuning training data via fine-tuned teacher LLMs to improve the downstream performance of much smaller models. In four text classification and two text generation tasks, we find that both data generation and annotation dramatically improve the respective downstream model's performance, occasionally necessitating only a minor fraction of the original training dataset.