I Hear Your True Colors: Image Guided Audio Generation

TL;DR

Im2Wav is a novel image-guided audio generation system that leverages hierarchical Transformer models and CLIP embeddings to produce semantically relevant high-fidelity sounds from images, outperforming baselines.

Contribution

The paper introduces Im2Wav, combining hierarchical audio modeling with CLIP-based visual conditioning and a new benchmark dataset for image-to-audio tasks.

Findings

Outperforms baseline models in fidelity and relevance

Uses CLIP embeddings for effective visual conditioning

Provides an ablation study and a new evaluation dataset

Abstract

We propose Im2Wav, an image guided open-domain audio generation system. Given an input image or a sequence of images, Im2Wav generates a semantically relevant sound. Im2Wav is based on two Transformer language models, that operate over a hierarchical discrete audio representation obtained from a VQ-VAE based model. We first produce a low-level audio representation using a language model. Then, we upsample the audio tokens using an additional language model to generate a high-fidelity audio sample. We use the rich semantics of a pre-trained CLIP (Contrastive Language-Image Pre-training) embedding as a visual representation to condition the language model. In addition, to steer the generation process towards the conditioning image, we apply the classifier-free guidance method. Results suggest that Im2Wav significantly outperforms the evaluated baselines in both fidelity and relevance evaluation metrics. Additionally, we provide an ablation study to better assess the impact of each of the method components on overall performance. Lastly, to better evaluate image-to-audio models, we propose an out-of-domain image dataset, denoted as ImageHear. ImageHear can be used as a benchmark for evaluating future image-to-audio models. Samples and code can be found inside the manuscript.