Towards General Auditory Intelligence: Large Multimodal Models for Machine Listening and Speaking

TL;DR

This paper reviews recent advances in large multimodal models for machine listening and speaking, emphasizing audio comprehension, generation, speech interaction, and audio-visual understanding to move towards general auditory intelligence.

Contribution

It provides a comprehensive survey of integrating audio into large language models, highlighting recent progress, challenges, and future directions for audio-native AGI systems.

Findings

LLMs are transforming audio perception and reasoning.

Multimodal fusion enhances situational awareness.

Current challenges include deep semantic understanding and natural interaction.

Abstract

In the era of large language models (LLMs) and artificial general intelligence (AGI), computer audition must evolve beyond traditional paradigms to fully leverage the capabilities of foundation models, towards more comprehensive understanding, more natural generation and more human-like interaction. Audio, as a modality rich in semantic, emotional, and contextual cues, plays a vital role in achieving naturalistic and embodied machine intelligence. This survey provides a comprehensive review of recent progress in integrating audio into LLMs, with a focus on four key areas: audio comprehension, audio generation, speech-based interaction, and audio-visual understanding. We analyze how LLMs are reshaping audio perception and reasoning, enabling systems to understand sound at a deeper semantic level, generate expressive audio outputs, and engage in human-like spoken interaction. Furthermore, we explore how the fusion of audio and visual modalities enhances situational awareness and cross-modal reasoning, pushing the boundaries of multimodal intelligence. This survey not only synthesizes existing research but also identifies critical challenges and future directions for building audio-native AGI systems capable of perceiving, understanding, and interacting through sound as naturally as humans do.