Dial-In LLM: Human-Aligned LLM-in-the-loop Intent Clustering for Customer Service Dialogues

TL;DR

This paper introduces an LLM-in-the-loop framework for intent clustering in customer service dialogues, improving semantic coherence, cluster naming, and discovering intent clusters effectively, supported by a large Chinese dataset.

Contribution

It presents a novel LLM-in-the-loop approach integrating LLMs into clustering, with new techniques for semantic evaluation, cluster naming, and a large Chinese dataset for dialogue intent.

Findings

Achieved over 95% accuracy in semantic coherence evaluation.

Significantly outperformed baseline clustering methods.

Enhanced downstream application performance.

Abstract

Discovering customer intentions is crucial for automated service agents, yet existing intent clustering methods often fall short due to their reliance on embedding distance metrics and neglect of underlying semantic structures. To address these limitations, we propose an LLM-in-the-loop (LLM-ITL) intent clustering framework, integrating the language understanding capabilities of LLMs into conventional clustering algorithms. Specifically, this paper (1) examines the effectiveness of fine-tuned LLMs in semantic coherence evaluation and intent cluster naming, achieving over 95% accuracy aligned with human judgments; (2) designs an LLM-ITL framework that facilitates the iterative discovery of coherent intent clusters and the optimal number of clusters; and (3) introduces context-aware techniques tailored for customer service dialogue. Since existing English benchmarks lack sufficient semantic diversity and intent coverage, we further present a comprehensive Chinese dialogue intent dataset comprising over 100k real customer service calls with 1,507 human-annotated clusters. The proposed approaches significantly outperform LLM-guided baselines, achieving notable improvements in clustering quality, cost efficiency, and downstream applications. Combined with several best practices, our findings highlight the prominence of LLM-in-the-loop techniques for scalable dialogue data mining.