Minimally Supervised Hierarchical Domain Intent Learning for CRS

TL;DR

This paper presents a neural attention-based hierarchical clustering method that efficiently models evolving domain intents in conversational recommendation systems, reducing data requirements and enabling scalable, adaptive intent management.

Contribution

It introduces a minimal-supervision hierarchical clustering algorithm that leverages neural attention to optimize intent grouping with less data, improving CRS scalability.

Findings

Reduces the number of questions needed for intent coverage

Enhances scalability and adaptability of CRS

Demonstrates effectiveness on a 44,000-question dataset

Abstract

Modeling domain intent within an evolving domain structure presents a significant challenge for domain-specific conversational recommendation systems (CRS). The conventional approach involves training an intent model using utterance-intent pairs. However, as new intents and patterns emerge, the model must be continuously updated while preserving existing relationships and maintaining efficient retrieval. This process leads to substantial growth in utterance-intent pairs, making manual labeling increasingly costly and impractical. In this paper, we propose an efficient solution for constructing a dynamic hierarchical structure that minimizes the number of user utterances required to achieve adequate domain knowledge coverage. To this end, we introduce a neural network-based attention-driven hierarchical clustering algorithm designed to optimize intent grouping using minimal data. The proposed method builds upon and integrates concepts from two existing flat clustering algorithms DEC and NAM, both of which utilize neural attention mechanisms. We apply our approach to a curated subset of 44,000 questions from the business food domain. Experimental results demonstrate that constructing the hierarchy using a stratified sampling strategy significantly reduces the number of questions needed to represent the evolving intent structure. Our findings indicate that this approach enables efficient coverage of dynamic domain knowledge without frequent retraining, thereby enhancing scalability and adaptability in domain-specific CSRs.