When Audio-LLMs Don't Listen: A Cross-Linguistic Study of Modality Arbitration

TL;DR

This study investigates how speech-enabled language models prioritize text over audio when conflicting information arises, revealing a bias towards text and exploring factors influencing arbitration behavior across multiple languages and models.

Contribution

Introduces ALME, a large dataset for cross-linguistic audio-text conflict evaluation, and analyzes arbitration behavior and biases in state-of-the-art audio-LLMs.

Findings

Models follow text 10-26 times more often than audio in conflicts.

Framing transcripts as corrupted reduces model reliance on text.

Arbitration behavior depends more on reasoning than on audio input quality.

Abstract

When audio and text conflict, speech-enabled language models follow text far more often than they do when arbitrating between two conflicting text sources, even under explicit instructions to trust the audio. We introduce ALME (Audio-LLM Modality Evaluation), a dataset of 57,602 controlled audio-text conflict stimuli across eight languages, together with Text Dominance Ratio (TDR), which measures how often a model follows conflicting text when instructed to follow audio. Gemini 2.0 Flash and GPT-4o show TDR 10--26$\times$ higher than a baseline that replaces audio with its transcript under otherwise identical conditions (Gemini 2.0 Flash: 16.6% vs. 1.6%; GPT-4o: 23.2% vs. 0.9%). These results suggest that text dominance reflects not only information content, but also an asymmetry in arbitration accessibility, i.e., how easily the model can use competing representations at decision time. Framing the transcript as deliberately corrupted reduces TDR by 80%, whereas forcing explicit transcription increases it by 14%. A fine-tuning ablation further suggests that arbitration behavior depends more on LLM reasoning than on the audio input path alone. Across four audio-LLMs, we observe the same qualitative pattern with substantial cross-model and cross-linguistic variation.