MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark

TL;DR

MMSU is a new comprehensive benchmark with 5,000 audio-question-answer triplets across 47 tasks, designed to evaluate and advance multi-faceted spoken language understanding and reasoning in Speech Large Language Models.

Contribution

This paper introduces MMSU, a large-scale benchmark for evaluating speech understanding and reasoning, incorporating diverse linguistic phenomena and providing a standard for future model development.

Findings

Existing SpeechLLMs show significant room for improvement.

MMSU covers a wide range of linguistic phenomena.

Evaluation highlights the need for further model optimization.

Abstract

Speech inherently contains rich acoustic information that extends far beyond the textual language. In real-world spoken language understanding, effective interpretation often requires integrating semantic meaning (e.g., content), paralinguistic features (e.g., emotions, speed, pitch) and phonological characteristics (e.g., prosody, intonation, rhythm), which are embedded in speech. While recent multimodal Speech Large Language Models (SpeechLLMs) have demonstrated remarkable capabilities in processing audio information, their ability to perform fine-grained perception and complex reasoning in natural speech remains largely unexplored. To address this gap, we introduce MMSU, a comprehensive benchmark designed specifically for understanding and reasoning in spoken language. MMSU comprises 5,000 meticulously curated audio-question-answer triplets across 47 distinct tasks. To ground our benchmark in linguistic theory, we systematically incorporate a wide range of linguistic phenomena, including phonetics, prosody, rhetoric, syntactics, semantics, and paralinguistics. Through a rigorous evaluation of 14 advanced SpeechLLMs, we identify substantial room for improvement in existing models, highlighting meaningful directions for future optimization. MMSU establishes a new standard for comprehensive assessment of spoken language understanding, providing valuable insights for developing more sophisticated human-AI speech interaction systems. MMSU benchmark is available at https://huggingface.co/datasets/ddwang2000/MMSU. Evaluation Code is available at https://github.com/dingdongwang/MMSU.