NATURE: Natural Auxiliary Text Utterances for Realistic Spoken Language Evaluation

TL;DR

This paper introduces NATURE, a set of transformations that simulate human spoken language variations to evaluate the robustness of slot-filling and intent detection models, revealing significant performance drops on standard benchmarks.

Contribution

NATURE provides a simple, effective method to test the generalization of spoken language understanding models under realistic spoken language variations.

Findings

Model accuracy drops by up to 40% with NATURE transformations.

Standard benchmarks may overestimate model robustness in real-world scenarios.

Simple perturbations reveal vulnerabilities in current spoken language models.

Abstract

Slot-filling and intent detection are the backbone of conversational agents such as voice assistants, and are active areas of research. Even though state-of-the-art techniques on publicly available benchmarks show impressive performance, their ability to generalize to realistic scenarios is yet to be demonstrated. In this work, we present NATURE, a set of simple spoken-language oriented transformations, applied to the evaluation set of datasets, to introduce human spoken language variations while preserving the semantics of an utterance. We apply NATURE to common slot-filling and intent detection benchmarks and demonstrate that simple perturbations from the standard evaluation set by NATURE can deteriorate model performance significantly. Through our experiments we demonstrate that when NATURE operators are applied to evaluation set of popular benchmarks the model accuracy can drop by up to 40%.