Fleurs-SLU: A Massively Multilingual Benchmark for Spoken Language Understanding

TL;DR

Fleurs-SLU introduces a comprehensive multilingual spoken language understanding benchmark with extensive speech data across over 100 languages, enabling evaluation of various models and revealing insights into the effectiveness of cascaded, end-to-end, and multimodal approaches.

Contribution

The paper presents Fleurs-SLU, the first large-scale multilingual SLU benchmark covering 102 languages, and provides extensive evaluation of different SLU systems on this dataset.

Findings

Cascaded systems are more robust in multilingual SLU.

Pretrained speech encoders perform competitively in topical speech classification.

Closed-source speech-LLMs can outperform cascaded systems.

Abstract

Spoken language understanding (SLU) is indispensable for half of all living languages that lack a formal writing system. Unlike for high-resource languages, for these languages, we cannot offload semantic understanding of speech to the cascade of automatic speech recognition (ASR) and text-based large language models (LLMs). Even if low-resource languages possess a writing system, ASR for these languages remains unreliable due to limited bimodal speech and text training data. Nonetheless, the evaluation of multilingual SLU is limited to shallow tasks such as intent classification or language identification. This is why we present Fleurs-SLU, a multilingual SLU benchmark that encompasses (i) 692 hours of speech for topical utterance classification in 102 languages and (ii) multiple-choice question answering via listening comprehension spanning 944 hours of speech across 92 languages. We extensively evaluate end-to-end speech classification models, cascaded systems that combine speech-to-text transcription with subsequent LLM-based classification, and multimodal speech-LLMs on Fleurs-SLU. Our results show that cascaded systems are more robust in multilingual SLU, though well-pretrained speech encoders can perform competitively in topical speech classification. Closed-source speech-LLMs match or surpass the performance of cascaded systems. We observe a strong correlation between robust multilingual ASR, effective speech-to-text translation, and strong multilingual SLU, indicating mutual benefits between acoustic and semantic speech representations.