All That Glitters Is Not Audio: Rethinking Text Priors and Audio Reliance in Audio-Language Evaluation

TL;DR

This paper introduces a diagnostic framework revealing that many audio-language models rely heavily on text priors, often achieving high scores without true audio understanding, thus questioning benchmark validity.

Contribution

It proposes a two-axis diagnostic approach to evaluate models' reliance on text priors and actual audio, highlighting the need for more reliable audio understanding assessments.

Findings

Models retain 60-72% of scores without audio input.

Only 3.0-4.2% of items require full audio clips.

Most items can be answered using localized audio fragments.

Abstract

Large Audio-Language Models show consistent performance gains across speech and audio benchmarks, yet high scores may not reflect true auditory perception. If a model can answer questions without processing the acoustic signal, the benchmark fails as a measure of auditory understanding. We present a diagnostic framework using two axes: text prior, which measures answerability from text and general knowledge alone, and audio reliance, which assesses actual dependency on the acoustic signal. Evaluating eight LALMs across three benchmarks, we find that models retain 60-72% of their full audio scores even without any audio input. Moreover, among items that require audio, only 3.0-4.2% need the complete audio clip; the majority can be resolved using localized fragments. These findings challenge the assumption that benchmark performance equals robust audio understanding, and we conclude with practical guidelines for improving evaluation reliability and benchmark design.