Dialog2Flow: Pre-training Soft-Contrastive Action-Driven Sentence Embeddings for Automatic Dialog Flow Extraction

TL;DR

Dialog2Flow introduces a novel pre-trained embedding method that maps dialog utterances into a latent space based on their communicative actions, enabling automatic extraction of dialog workflows from unannotated data.

Contribution

The paper presents a new soft-contrastive pre-training approach for action-driven sentence embeddings, specifically designed for dialog workflow extraction, with a comprehensive dataset and improved performance.

Findings

D2F embeddings outperform standard sentence embeddings in dialog tasks.

The soft contrastive loss effectively leverages action semantics.

Workflow extraction from dialogs is significantly improved.

Abstract

Efficiently deriving structured workflows from unannotated dialogs remains an underexplored and formidable challenge in computational linguistics. Automating this process could significantly accelerate the manual design of workflows in new domains and enable the grounding of large language models in domain-specific flowcharts, enhancing transparency and controllability. In this paper, we introduce Dialog2Flow (D2F) embeddings, which differ from conventional sentence embeddings by mapping utterances to a latent space where they are grouped according to their communicative and informative functions (i.e., the actions they represent). D2F allows for modeling dialogs as continuous trajectories in a latent space with distinct action-related regions. By clustering D2F embeddings, the latent space is quantized, and dialogs can be converted into sequences of region/action IDs, facilitating the extraction of the underlying workflow. To pre-train D2F, we build a comprehensive dataset by unifying twenty task-oriented dialog datasets with normalized per-turn action annotations. We also introduce a novel soft contrastive loss that leverages the semantic information of these actions to guide the representation learning process, showing superior performance compared to standard supervised contrastive loss. Evaluation against various sentence embeddings, including dialog-specific ones, demonstrates that D2F yields superior qualitative and quantitative results across diverse domains.