ARTA: Collection and Classification of Ambiguous Requests and Thoughtful Actions

TL;DR

This paper presents ARTA, a dataset and model for classifying ambiguous user requests into appropriate system actions, using positive/unlabeled learning to handle incomplete annotations and improve classification accuracy.

Contribution

The paper introduces a novel corpus and applies PU learning for classifying ambiguous requests, addressing annotation challenges in dialogue systems.

Findings

PU learning outperforms PN learning in classification accuracy.

The corpus enables better understanding of ambiguous user requests.

The model effectively suggests thoughtful system actions.

Abstract

Human-assisting systems such as dialogue systems must take thoughtful, appropriate actions not only for clear and unambiguous user requests, but also for ambiguous user requests, even if the users themselves are not aware of their potential requirements. To construct such a dialogue agent, we collected a corpus and developed a model that classifies ambiguous user requests into corresponding system actions. In order to collect a high-quality corpus, we asked workers to input antecedent user requests whose pre-defined actions could be regarded as thoughtful. Although multiple actions could be identified as thoughtful for a single user request, annotating all combinations of user requests and system actions is impractical. For this reason, we fully annotated only the test data and left the annotation of the training data incomplete. In order to train the classification model on such training data, we applied the positive/unlabeled (PU) learning method, which assumes that only a part of the data is labeled with positive examples. The experimental results show that the PU learning method achieved better performance than the general positive/negative (PN) learning method to classify thoughtful actions given an ambiguous user request.