DEAF: A Benchmark for Diagnostic Evaluation of Acoustic Faithfulness in Audio Language Models

TL;DR

This paper introduces DEAF, a benchmark to evaluate whether audio language models genuinely understand acoustic signals or mainly rely on textual cues, revealing a dominance of text over acoustic features in model predictions.

Contribution

The paper presents DEAF, a comprehensive benchmark and evaluation framework to systematically assess acoustic faithfulness in Audio MLLMs, highlighting their reliance on textual rather than acoustic information.

Findings

Models are sensitive to acoustic variations but mainly driven by textual cues.

High performance on standard benchmarks does not equate to genuine acoustic understanding.

DEAF reveals a significant gap between model capabilities and true acoustic processing.

Abstract

Recent Audio Multimodal Large Language Models (Audio MLLMs) demonstrate impressive performance on speech benchmarks, yet it remains unclear whether these models genuinely process acoustic signals or rely on text-based semantic inference. To systematically study this question, we introduce DEAF (Diagnostic Evaluation of Acoustic Faithfulness), a benchmark of over 2,700 conflict stimuli spanning three acoustic dimensions: emotional prosody, background sounds, and speaker identity. Then, we design a controlled multi-level evaluation framework that progressively increases textual influence, ranging from semantic conflicts in the content to misleading prompts and their combination, allowing us to disentangle content-driven bias from prompt-induced sycophancy. We further introduce diagnostic metrics to quantify model reliance on textual cues over acoustic signals. Our evaluation of seven Audio MLLMs reveals a consistent pattern of text dominance: models are sensitive to acoustic variations, yet predictions are predominantly driven by textual inputs, revealing a gap between high performance on standard speech benchmarks and genuine acoustic understanding.