MAC-SLU: Multi-Intent Automotive Cabin Spoken Language Understanding Benchmark

TL;DR

This paper introduces MAC-SLU, a challenging multi-intent spoken language understanding dataset for automotive cabins, and benchmarks various large language and audio models, revealing current limitations and potential of different approaches.

Contribution

The paper presents MAC-SLU, a new complex dataset for automotive SLU, and provides a comprehensive benchmark of LLMs and LALMs, highlighting their strengths and weaknesses.

Findings

LLMs perform well with in-context learning but lag behind supervised fine-tuning.

End-to-end LALMs match pipeline approaches and reduce error propagation.

The dataset increases SLU task difficulty with authentic multi-intent data.

Abstract

Spoken Language Understanding (SLU), which aims to extract user semantics to execute downstream tasks, is a crucial component of task-oriented dialog systems. Existing SLU datasets generally lack sufficient diversity and complexity, and there is an absence of a unified benchmark for the latest Large Language Models (LLMs) and Large Audio Language Models (LALMs). This work introduces MAC-SLU, a novel Multi-Intent Automotive Cabin Spoken Language Understanding Dataset, which increases the difficulty of the SLU task by incorporating authentic and complex multi-intent data. Based on MAC-SLU, we conducted a comprehensive benchmark of leading open-source LLMs and LALMs, covering methods like in-context learning, supervised fine-tuning (SFT), and end-to-end (E2E) and pipeline paradigms. Our experiments show that while LLMs and LALMs have the potential to complete SLU tasks through in-context learning, their performance still lags significantly behind SFT. Meanwhile, E2E LALMs demonstrate performance comparable to pipeline approaches and effectively avoid error propagation from speech recognition. Code\footnote{https://github.com/Gatsby-web/MAC\_SLU} and datasets\footnote{huggingface.co/datasets/Gatsby1984/MAC\_SLU} are released publicly.