A Survey on Speech Large Language Models for Understanding

TL;DR

This survey systematically reviews Speech Large Language Models, defining speech understanding, analyzing architectures, training strategies, datasets, and evaluation methods, while highlighting key challenges and future directions for robust, generalizable speech comprehension systems.

Contribution

It provides a formal definition and taxonomy of speech understanding, analyzes current Speech LLM architectures, and discusses challenges and directions for future research.

Findings

Identifies instruction sensitivity as a key challenge.

Highlights degradation in semantic reasoning in Speech LLMs.

Provides a structured framework for analyzing Speech LLM architectures.

Abstract

Speech understanding is essential for interpreting the diverse forms of information embedded in spoken language, including linguistic, paralinguistic, and non-linguistic cues that are vital for effective human-computer interaction. The rapid advancement of large language models (LLMs) has catalyzed the emergence of Speech Large Language Models (Speech LLMs), which marks a transformative shift toward general-purpose speech understanding systems. To further clarify and systematically delineate task objectives, in this paper, we formally define the concept of speech understanding and introduce a structured taxonomy encompassing its informational, functional, and format dimensions. Within this scope of definition, we present a comprehensive review of current Speech LLMs, analyzing their architectures through a three-stage abstraction: Modality Feature Extraction, Modality Information Fusion, and LLM Inference. In addition, we examine training strategies, discuss representative datasets, and review evaluation methodologies adopted in the field. Based on empirical analyses and experimental evidence, we identify two key challenges currently facing Speech LLMs: instruction sensitivity and degradation in semantic reasoning and propose concrete directions for addressing these issues. Through this systematic and detailed survey, we aim to offer a foundational reference for researchers and practitioners working toward more robust, generalizable, and human-aligned Speech LLMs.