A Single Example Can Improve Zero-Shot Data Generation

TL;DR

This paper demonstrates that a single example can significantly enhance zero-shot data generation for intent classification, enabling the creation of high-quality datasets with minimal data.

Contribution

It introduces a novel approach where a single example improves zero-shot data generation, reducing the need for extensive dataset collection.

Findings

Generated data closely matches original test sets

Zero-shot approach effectively generates unseen intent utterances

Single-example training enhances data quality

Abstract

Sub-tasks of intent classification, such as robustness to distribution shift, adaptation to specific user groups and personalization, out-of-domain detection, require extensive and flexible datasets for experiments and evaluation. As collecting such datasets is time- and labor-consuming, we propose to use text generation methods to gather datasets. The generator should be trained to generate utterances that belong to the given intent. We explore two approaches to generating task-oriented utterances. In the zero-shot approach, the model is trained to generate utterances from seen intents and is further used to generate utterances for intents unseen during training. In the one-shot approach, the model is presented with a single utterance from a test intent. We perform a thorough automatic, and human evaluation of the dataset generated utilizing two proposed approaches. Our results reveal that the attributes of the generated data are close to original test sets, collected via crowd-sourcing.