AutoGeTS: Knowledge-based Automated Generation of Text Synthetics for Improving Text Classification

TL;DR

This paper presents AutoGeTS, a method that uses large language models to generate synthetic text data, improving text classification performance without additional real data collection.

Contribution

It introduces an automated workflow that searches for input examples to generate more effective synthetic data and employs an ensemble strategy to select the best search method.

Findings

Ensemble approach outperforms individual search strategies.

Synthetic data improves classification accuracy.

Automated workflow reduces data collection efforts.

Abstract

When developing text classification models for real world applications, one major challenge is the difficulty to collect sufficient data for all text classes. In this work, we address this challenge by utilizing large language models (LLMs) to generate synthetic data and using such data to improve the performance of the models without waiting for more real data to be collected and labelled. As an LLM generates different synthetic data in response to different input examples, we formulate an automated workflow, which searches for input examples that lead to more ``effective'' synthetic data for improving the model concerned. We study three search strategies with an extensive set of experiments, and use experiment results to inform an ensemble algorithm that selects a search strategy according to the characteristics of a class. Our further experiments demonstrate that this ensemble approach is more effective than each individual strategy in our automated workflow for improving classification models using LLMs.