Investigating Cost-Efficiency of LLM-Generated Training Data for Conversational Semantic Frame Analysis

TL;DR

This paper explores how to balance cost and quality in training data for conversational semantic analysis by combining human and GPT-4 generated data, optimizing performance across different budgets.

Contribution

It introduces a method for allocating budgets between human and LLM-generated data to maximize cost-efficiency in training models.

Findings

Optimal performance is achieved by combining human and LLM data.

Higher proportions of LLM data are preferable at lower budgets.

Mixing data sources improves cost-efficiency across various budget levels.

Abstract

Recent studies have demonstrated that few-shot learning allows LLMs to generate training data for supervised models at a low cost. However, the quality of LLM-generated data may not entirely match that of human-labeled data. This raises a crucial question: how should one balance the trade-off between the higher quality but more expensive human data and the lower quality yet substantially cheaper LLM-generated data? In this paper, we synthesized training data for conversational semantic frame analysis using GPT-4 and examined how to allocate budgets optimally to achieve the best performance. Our experiments, conducted across various budget levels, reveal that optimal cost-efficiency is achieved by combining both human and LLM-generated data across a wide range of budget levels. Notably, as the budget decreases, a higher proportion of LLM-generated data becomes more preferable.