Discovering New Intents Using Latent Variables

TL;DR

This paper introduces a probabilistic framework using latent variables and Expectation Maximization to discover new intents in dialogue systems, effectively leveraging unlabeled data and preserving prior knowledge.

Contribution

It proposes a novel intent discovery method that models intent assignments as latent variables and balances knowledge transfer with new intent identification.

Findings

Achieves significant improvements on three real-world datasets.

Effectively explores the intrinsic structure of unlabeled data.

Reduces forgetting of prior knowledge during intent discovery.

Abstract

Discovering new intents is of great significance to establishing Bootstrapped Task-Oriented Dialogue System. Most existing methods either lack the ability to transfer prior knowledge in the known intent data or fall into the dilemma of forgetting prior knowledge in the follow-up. More importantly, these methods do not deeply explore the intrinsic structure of unlabeled data, so they can not seek out the characteristics that make an intent in general. In this paper, starting from the intuition that discovering intents could be beneficial to the identification of the known intents, we propose a probabilistic framework for discovering intents where intent assignments are treated as latent variables. We adopt Expectation Maximization framework for optimization. Specifically, In E-step, we conduct discovering intents and explore the intrinsic structure of unlabeled data by the posterior of intent assignments. In M-step, we alleviate the forgetting of prior knowledge transferred from known intents by optimizing the discrimination of labeled data. Extensive experiments conducted in three challenging real-world datasets demonstrate our method can achieve substantial improvements.