Seeing Sound: Assembling Sounds from Visuals for Audio-to-Image Generation

TL;DR

This paper introduces a scalable framework for generating audio-to-image models by pairing disjoint uni-modal data through retrieval and reasoning, enabling diverse training data without strict audio-visual correspondence.

Contribution

It proposes a novel image sonification method that leverages vision-language models to artificially pair uni-modal data, improving data diversity for audio-to-image generation.

Findings

The approach achieves competitive results with state-of-the-art models.

The model demonstrates capabilities like semantic mixing and interpolation.

It exhibits auditory features such as loudness calibration and reverberation modeling.

Abstract

Training audio-to-image generative models requires an abundance of diverse audio-visual pairs that are semantically aligned. Such data is almost always curated from in-the-wild videos, given the cross-modal semantic correspondence that is inherent to them. In this work, we hypothesize that insisting on the absolute need for ground truth audio-visual correspondence, is not only unnecessary, but also leads to severe restrictions in scale, quality, and diversity of the data, ultimately impairing its use in the modern generative models. That is, we propose a scalable image sonification framework where instances from a variety of high-quality yet disjoint uni-modal origins can be artificially paired through a retrieval process that is empowered by reasoning capabilities of modern vision-language models. To demonstrate the efficacy of this approach, we use our sonified images to train an audio-to-image generative model that performs competitively against state-of-the-art. Finally, through a series of ablation studies, we exhibit several intriguing auditory capabilities like semantic mixing and interpolation, loudness calibration and acoustic space modeling through reverberation that our model has implicitly developed to guide the image generation process.