On the Robustness of Intent Classification and Slot Labeling in Goal-oriented Dialog Systems to Real-world Noise

TL;DR

This paper evaluates the robustness of intent classification and slot labeling models in goal-oriented dialogue systems against real-world noise, and proposes data augmentation techniques to improve their resilience across multiple noise types.

Contribution

It introduces a comprehensive noise test-suite and a novel data augmentation approach that enhances model robustness to diverse real-world noise in dialogue systems.

Findings

Noise significantly degrades model performance.

Data augmentation improves robustness by over 10% in accuracy.

First model to be robust across multiple noise types.

Abstract

Intent Classification (IC) and Slot Labeling (SL) models, which form the basis of dialogue systems, often encounter noisy data in real-word environments. In this work, we investigate how robust IC/SL models are to noisy data. We collect and publicly release a test-suite for seven common noise types found in production human-to-bot conversations (abbreviations, casing, misspellings, morphological variants, paraphrases, punctuation and synonyms). On this test-suite, we show that common noise types substantially degrade the IC accuracy and SL F1 performance of state-of-the-art BERT-based IC/SL models. By leveraging cross-noise robustness transfer -- training on one noise type to improve robustness on another noise type -- we design aggregate data-augmentation approaches that increase the model performance across all seven noise types by +10.8% for IC accuracy and +15 points for SL F1 on average. To the best of our knowledge, this is the first work to present a single IC/SL model that is robust to a wide range of noise phenomena.