RSA-Bench: Benchmarking Audio Large Models in Real-World Acoustic Scenarios

TL;DR

RSA-Bench introduces a comprehensive auditory scene simulation benchmark to evaluate Audio Large Models' robustness in complex, real-world acoustic environments across multiple tasks.

Contribution

It presents a novel evaluation framework that uses natural environmental soundscapes to stress-test ALMs, revealing their strengths and vulnerabilities in realistic scenarios.

Findings

Models are resilient in low-level recognition but struggle with high-order reasoning under stress.

Vocal-like interference significantly disrupts model performance more than mechanical noise.

Standard speech denoising can worsen model performance due to semantic distortions.

Abstract

While Audio Large Models (ALMs) have achieved remarkable proficiency, their robustness remains brittle in real-world deployment. Existing evaluations largely rely on synthetic Gaussian noise or simplistic single-source interference, failing to capture the intricate, multi-layered acoustic dynamics -- or ``Acoustic Ecology'' -- that characterize authentic physical environments. To bridge this ecological gap, we introduce \textbf{RSA-Bench}, a comprehensive robustness benchmark designed to stress-test ALLMs through high-fidelity auditory scene simulations. Unlike traditional methods, we construct evaluation samples by naturally superimposing diverse environmental soundscapes -- spanning \textit{Pasture}, \textit{Extreme Weather}, \textit{Classroom}, and \textit{Outdoors} -- onto clean speech signals across a spectrum of interference intensities. By evaluating models on six core tasks ranging from fundamental perception to complex reasoning, our study unveils three macro-level insights: \textbf{(I) The Perception-Cognition Gap:} Models maintain relative resilience in low-level recognition but suffer a \textbf{functional collapse} in high-order reasoning tasks under stress; \textbf{(II) Scenario Sensitivity:} ``Vocal-like'' interference (e.g., background laughter) proves significantly more destructive than mechanical noise, challenging the model's auditory attention mechanisms; and \textbf{(III) The Denoising Paradox:} Standard speech enhancement often exacerbates performance degradation, as ALLMs prove highly sensitive to the semantic distortions introduced by denoising artifacts.