Can Sound Replace Vision in LLaVA With Token Substitution?

TL;DR

This paper investigates the effects of extreme audio-visual alignment on perceptual models by creating a detailed dataset and analyzing how different encoder architectures respond to realignment in the CLIP space.

Contribution

It introduces a new dataset with granular alignment scores and systematically studies how image-centric and text-centric encoders behave under superaligned audio-visual representations.

Findings

Image-centric encoders excel in cross-modal retrieval but lose linguistic detail after alignment.

Text-centric encoders better preserve linguistic information during alignment.

Alignment impacts encoder performance differently based on their architectural design.

Abstract

What happens when we push audio-visual alignment to its absolute limits? To systematically investigate this question, we needed datasets with granular alignment quality annotations, but existing datasets treat alignment as binary, either synchronized or not. To address this limitation, we developed a comprehensive dataset featuring detailed alignment scores that reveal the hidden spectrum of audio-visual perceptual correspondence. Using these precise scores, we create "superaligned" representations by training exclusively on the most perfectly matched audio-visual pairs, then conduct our systematic investigation into how this extreme alignment transforms perceptual model behavior across retrieval and generation tasks. The encoders under study fall into two main groups consisting of image-centric encoders that were pretrained using visual modalities as intermediary hubs for connecting modalities, and text-centric encoders that were pretrained with direct audio-language alignment. We first measure the baseline performance of these encoders on two key tasks, namely cross-modal retrieval and text description generation in vision-language models. Subsequently, we realign all encoders with the CLIP space using highly coherent audio-visual data and observe the performance changes. Our findings reveal that the initial architectural type of the encoder determines how it responds to the alignment process. Image-centric encoders, which are inherently designed for alignment, demonstrate exceptional performance in cross-modal retrieval, but this intensive alignment causes compression of unique linguistic information and reduces the quality of their text description generation in vision-language models. In contrast, text-centric encoders, which possess stronger linguistic authenticity, are able to maintain a better balance between the two objectives.