Gaining Insights into Unrecognized User Utterances in Task-Oriented Dialog Systems

TL;DR

This paper introduces an end-to-end pipeline for analyzing unrecognized user utterances in goal-oriented dialog systems, including clustering, representative extraction, and naming, to improve system understanding and performance.

Contribution

It presents a novel, tailored clustering and analysis pipeline specifically designed for unrecognized utterances in real-world dialog systems, enhancing intent recognition insights.

Findings

Effective clustering of unrecognized requests

Improved understanding of user intent gaps

Enhanced system performance through analysis

Abstract

The rapidly growing market demand for automatic dialogue agents capable of goal-oriented behavior has caused many tech-industry leaders to invest considerable efforts into task-oriented dialog systems. The success of these systems is highly dependent on the accuracy of their intent identification -- the process of deducing the goal or meaning of the user's request and mapping it to one of the known intents for further processing. Gaining insights into unrecognized utterances -- user requests the systems fail to attribute to a known intent -- is therefore a key process in continuous improvement of goal-oriented dialog systems. We present an end-to-end pipeline for processing unrecognized user utterances, deployed in a real-world, commercial task-oriented dialog system, including a specifically-tailored clustering algorithm, a novel approach to cluster representative extraction, and cluster naming. We evaluated the proposed components, demonstrating their benefits in the analysis of unrecognized user requests.