Seeing Sound, Hearing Sight: Uncovering Modality Bias and Conflict of AI models in Sound Localization

TL;DR

This paper investigates how AI models handle conflicting visual and auditory cues in sound localization, revealing a visual bias in current systems and proposing a new model that aligns more closely with human perception.

Contribution

The study systematically compares AI and human performance in audiovisual conflicts, introduces EchoPin, a neuroscience-inspired model, and demonstrates its superior performance and human-like bias.

Findings

Humans outperform AI in resolving audiovisual conflicts.

AI models tend to default to visual cues, reducing accuracy.

EchoPin surpasses existing benchmarks and exhibits human-like localization bias.

Abstract

Imagine hearing a dog bark and turning toward the sound only to see a parked car, while the real, silent dog sits elsewhere. Such sensory conflicts test perception, yet humans reliably resolve them by prioritizing sound over misleading visuals. Despite advances in multimodal AI integrating vision and audio, little is known about how these systems handle cross-modal conflicts or whether they favor one modality. In this study, we systematically examine modality bias and conflict resolution in AI sound localization. We assess leading multimodal models and benchmark them against human performance in psychophysics experiments across six audiovisual conditions, including congruent, conflicting, and absent cues. Humans consistently outperform AI, demonstrating superior resilience to conflicting or missing visuals by relying on auditory information. In contrast, AI models often default to visual input, degrading performance to near chance levels. To address this, we propose a neuroscience-inspired model, EchoPin, which uses a stereo audio-image dataset generated via 3D simulations. Even with limited training data, EchoPin surpasses existing benchmarks. Notably, it also mirrors human-like horizontal localization bias favoring left-right precision-likely due to the stereo audio structure reflecting human ear placement. These findings underscore how sensory input quality and system architecture shape multimodal representation accuracy.