MULTI3NLU++: A Multilingual, Multi-Intent, Multi-Domain Dataset for Natural Language Understanding in Task-Oriented Dialogue

TL;DR

This paper introduces MULTI3NLU++, a comprehensive multilingual dataset for natural language understanding in task-oriented dialogue, covering multiple languages, domains, and user intents to improve system generalization.

Contribution

The creation of MULTI3NLU++, the first multilingual, multi-intent, multi-domain dataset for NLU in task-oriented dialogue, including translations into diverse resource languages.

Findings

Multilingual models struggle with low-resource languages.

The dataset reveals the complexity of multi-domain, multi-intent NLU tasks.

Benchmark results highlight room for improvement in multilingual TOD systems.

Abstract

Task-oriented dialogue (TOD) systems have been widely deployed in many industries as they deliver more efficient customer support. These systems are typically constructed for a single domain or language and do not generalise well beyond this. To support work on Natural Language Understanding (NLU) in TOD across multiple languages and domains simultaneously, we constructed MULTI3NLU++, a multilingual, multi-intent, multi-domain dataset. MULTI3NLU++ extends the English only NLU++ dataset to include manual translations into a range of high, medium, and low resource languages (Spanish, Marathi, Turkish and Amharic), in two domains (BANKING and HOTELS). Because of its multi-intent property, MULTI3NLU++ represents complex and natural user goals, and therefore allows us to measure the realistic performance of TOD systems in a varied set of the world's languages. We use MULTI3NLU++ to benchmark state-of-the-art multilingual models for the NLU tasks of intent detection and slot labelling for TOD systems in the multilingual setting. The results demonstrate the challenging nature of the dataset, particularly in the low-resource language setting, offering ample room for future experimentation in multi-domain multilingual TOD setups.