HINT3: Raising the bar for Intent Detection in the Wild

TL;DR

This paper introduces three real-world chatbot datasets to improve intent detection benchmarking, revealing current systems' reliance on unintended patterns and highlighting the need for more robust models.

Contribution

The paper presents new real-user chatbot datasets and evaluates existing NLU systems, exposing their limitations in handling real-world intent detection complexities.

Findings

Current systems rely on unintended correlations in training data.

Performance saturates at low levels on real-world test sets.

Benchmarking with real datasets exposes robustness issues.

Abstract

Intent Detection systems in the real world are exposed to complexities of imbalanced datasets containing varying perception of intent, unintended correlations and domain-specific aberrations. To facilitate benchmarking which can reflect near real-world scenarios, we introduce 3 new datasets created from live chatbots in diverse domains. Unlike most existing datasets that are crowdsourced, our datasets contain real user queries received by the chatbots and facilitates penalising unwanted correlations grasped during the training process. We evaluate 4 NLU platforms and a BERT based classifier and find that performance saturates at inadequate levels on test sets because all systems latch on to unintended patterns in training data.