Chat Detection in an Intelligent Assistant: Combining Task-oriented and Non-task-oriented Spoken Dialogue Systems

TL;DR

This paper develops a dataset and methods for detecting whether users intend to chat with intelligent assistants, combining supervised learning with web data to improve accuracy.

Contribution

It introduces a new dataset of real user utterances and explores leveraging tweets and web queries for enhanced chat detection in hybrid dialogue systems.

Findings

Supervised methods achieve an F1-score of 86.21.

Adding tweets and web queries increases F1-score to 87.53.

Constructed a dataset of 15,160 utterances from real assistant logs.

Abstract

Recently emerged intelligent assistants on smartphones and home electronics (e.g., Siri and Alexa) can be seen as novel hybrids of domain-specific task-oriented spoken dialogue systems and open-domain non-task-oriented ones. To realize such hybrid dialogue systems, this paper investigates determining whether or not a user is going to have a chat with the system. To address the lack of benchmark datasets for this task, we construct a new dataset consisting of 15; 160 utterances collected from the real log data of a commercial intelligent assistant (and will release the dataset to facilitate future research activity). In addition, we investigate using tweets and Web search queries for handling open-domain user utterances, which characterize the task of chat detection. Experiments demonstrated that, while simple supervised methods are effective, the use of the tweets and search queries further improves the F1-score from 86.21 to 87.53.