U-SAM: An audio language Model for Unified Speech, Audio, and Music Understanding

TL;DR

U-SAM is a unified audio language model that effectively integrates speech, audio, and music understanding using specialized encoders, a large language model, and novel loss functions to improve cross-modal alignment and generalization.

Contribution

The paper introduces U-SAM, a novel audio language model that combines domain-specific encoders, a Mixture of Experts projector, and a Semantic-Aware Contrastive Loss for improved multi-domain audio understanding.

Findings

U-SAM outperforms existing models on multiple benchmarks.

It demonstrates strong generalization to unseen audio tasks.

The model achieves better cross-modal alignment through its loss design.

Abstract

The text generation paradigm for audio tasks has opened new possibilities for unified audio understanding. However, existing models face significant challenges in achieving a comprehensive understanding across diverse audio types, such as speech, general audio events, and music. Furthermore, their exclusive reliance on cross-entropy loss for alignment often falls short, as it treats all tokens equally and fails to account for redundant audio features, leading to weaker cross-modal alignment. To deal with the above challenges, this paper introduces U-SAM, an advanced audio language model that integrates specialized encoders for speech, audio, and music with a pre-trained large language model (LLM). U-SAM employs a Mixture of Experts (MoE) projector for task-aware feature fusion, dynamically routing and integrating the domain-specific encoder outputs. Additionally, U-SAM incorporates a Semantic-Aware Contrastive Loss Module, which explicitly identifies redundant audio features under language supervision and rectifies their semantic and spectral representations to enhance cross-modal alignment. Extensive experiments demonstrate that U-SAM consistently outperforms both specialized models and existing audio language models across multiple benchmarks. Moreover, it exhibits emergent capabilities on unseen tasks, showcasing its generalization potential. Code is available (https://github.com/Honee-W/U-SAM/).