Analysis of Utterance Embeddings and Clustering Methods Related to Intent Induction for Task-Oriented Dialogue

TL;DR

This paper investigates unsupervised methods for intent induction in task-oriented dialogue, emphasizing the importance of embedding and clustering choices, and demonstrates that MiniLM embeddings with Agglomerative clustering significantly improve performance.

Contribution

It systematically compares clustering algorithms and embeddings for intent induction, highlighting the effectiveness of MiniLM with Agglomerative clustering.

Findings

MiniLM with Agglomerative clustering outperforms other methods in key metrics.

Careful selection of embedding and clustering methods is crucial for intent induction.

The study provides insights into unsupervised intent labeling for dialogue systems.

Abstract

The focus of this work is to investigate unsupervised approaches to overcome quintessential challenges in designing task-oriented dialog schema: assigning intent labels to each dialog turn (intent clustering) and generating a set of intents based on the intent clustering methods (intent induction). We postulate there are two salient factors for automatic induction of intents: (1) clustering algorithm for intent labeling and (2) user utterance embedding space. We compare existing off-the-shelf clustering models and embeddings based on DSTC11 evaluation. Our extensive experiments demonstrate that the combined selection of utterance embedding and clustering method in the intent induction task should be carefully considered. We also present that pretrained MiniLM with Agglomerative clustering shows significant improvement in NMI, ARI, F1, accuracy and example coverage in intent induction tasks. The source codes are available at https://github.com/Jeiyoon/dstc11-track2.