Out-of-Scope Domain and Intent Classification through Hierarchical Joint Modeling

TL;DR

This paper introduces a hierarchical joint modeling approach for out-of-scope intent classification in dialog systems, effectively leveraging domain-intent correlations to improve accuracy and out-of-scope detection.

Contribution

It proposes a novel hierarchical multi-task learning model that jointly classifies domain and intent, replacing traditional two-stage methods and enhancing performance.

Findings

Outperforms existing methods in accuracy, recall, and F1 score.

Hierarchical model effectively captures domain-intent correlations.

Threshold-based post-processing further improves classification balance.

Abstract

User queries for a real-world dialog system may sometimes fall outside the scope of the system's capabilities, but appropriate system responses will enable smooth processing throughout the human-computer interaction. This paper is concerned with the user's intent, and focuses on out-of-scope intent classification in dialog systems. Although user intents are highly correlated with the application domain, few studies have exploited such correlations for intent classification. Rather than developing a two-stage approach that first classifies the domain and then the intent, we propose a hierarchical multi-task learning approach based on a joint model to classify domain and intent simultaneously. Novelties in the proposed approach include: (1) sharing supervised out-of-scope signals in joint modeling of domain and intent classification to replace a two-stage pipeline; and (2) introducing a hierarchical model that learns the intent and domain representations in the higher and lower layers respectively. Experiments show that the model outperforms existing methods in terms of accuracy, out-of-scope recall and F1. Additionally, threshold-based post-processing further improves performance by balancing precision and recall in intent classification.