Does Collaborative Human-LM Dialogue Generation Help Information Extraction from Human Dialogues?

TL;DR

This paper presents a human-in-the-loop dialogue generation framework that synthesizes realistic conversations to improve information extraction in private call center data, achieving significant performance gains.

Contribution

It introduces a novel dialogue synthesis method for privacy-sensitive domains and demonstrates its effectiveness in enhancing information extraction from real-world call center dialogues.

Findings

25% relative F1 improvement with synthetic data augmentation

Synthetic dialogues capture complex real-world call center interactions

Framework supports privacy-preserving data augmentation

Abstract

The capabilities of pretrained language models have opened opportunities to explore new application areas, but applications involving human-human interaction are limited by the fact that most data is protected from public release for privacy reasons. Problem-solving human dialogues in real applications can be much more complex than existing Wizard-of-Oz collections, preventing successful domain transfer. To support information extraction (IE) for a private call center dataset, we introduce a human-in-the-loop dialogue generation framework capable of synthesizing realistic dialogues. In IE experiments with auto insurance call center dialogues, we observe 25\% relative improvement in $F_1$ after augmenting a small set of real human conversations with synthetic data. We release code and our synthetic dataset to illustrate the complexity of real-world call center conversations and encourage development of complex dialogue datasets that are more representative of natural data.