TSCAN : Dialog Structure discovery using SCAN

TL;DR

This paper introduces TSCAN, an unsupervised method using SCAN and BERT to discover dialog structures by clustering utterances without relying on predefined ontologies, enabling interpretable dialog segmentation.

Contribution

The paper presents a novel unsupervised approach combining SCAN and BERT for dialog structure discovery, eliminating the need for labeled data or ontologies.

Findings

Clusters correspond to meaningful dialog segments

Transition probabilities reveal dialog flow patterns

Method achieves interpretable dialog structures without supervision

Abstract

Can we discover dialog structure by dividing utterances into labelled clusters. Can these labels be generated from the data. Typically for dialogs we need an ontology and use that to discover structure, however by using unsupervised classification and self-labelling we are able to intuit this structure without any labels or ontology. In this paper we apply SCAN (Semantic Clustering using Nearest Neighbors) to dialog data. We used BERT for pretext task and an adaptation of SCAN for clustering and self labeling. These clusters are used to identify transition probabilities and create the dialog structure. The self-labelling method used for SCAN makes these structures interpretable as every cluster has a label. As the approach is unsupervised, evaluation metrics is a challenge, we use statistical measures as proxies for structure quality