When Vision Speaks for Sound
Xiaofei Wen, Wenjie Jacky Mo, Xingyu Fu, Rui Cai, Tinghui Zhu, Wendi Li, Yanan Xie, Muhao Chen, Peng Qi

TL;DR
This paper reveals that current video-capable multimodal models often rely on visual cues to infer audio information rather than verifying audio streams, highlighting a vision-driven bias.
Contribution
The authors introduce Thud, a probing framework with counterfactual audio edits, and propose a two-stage alignment method to improve audio verification in multimodal models.
Findings
Models exploit visual-acoustic correlations without true audio verification.
The proposed method improves performance on intervention dimensions by 28 percentage points.
Slight improvements observed on general video and audio-visual QA benchmarks.
Abstract
Despite rapid progress in video-capable MLLMs, we find that their apparent audio understanding in videos is often vision-driven: models rely on visual cues to infer or hallucinate acoustic information, rather than verifying the audio stream. This issue appears across both state-of-the-art open-source omni models and leading closed-source models from providers such as Google and OpenAI. We characterize this failure mode as an audio-visual Clever Hans effect, in which models appear (falsely) audio-grounded, but actually exploit visual-acoustic correlations without verifying whether the audio and visual streams are truly aligned. To systematically study this behavior, we introduce Thud, an intervention-driven probing framework based on three counterfactual audio edits: Shift, which tests temporal synchronization; Mute, which tests sound existence; and Swap, which tests audio-visual…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
