Align, Adapt and Inject: Sound-guided Unified Image Generation

TL;DR

This paper introduces a unified framework that leverages sound to guide image generation, editing, and stylization by aligning audio with textual and visual representations, enhancing the capabilities of diffusion models.

Contribution

The proposed AAI framework effectively adapts sound into a token compatible with existing diffusion-based T2I models, enabling sound-guided image tasks with improved performance.

Findings

Outperforms state-of-the-art sound-guided image generation methods.

Achieves competitive results in audio-visual and audio-text retrieval tasks.

Enables flexible sound-guided image editing and stylization.

Abstract

Text-guided image generation has witnessed unprecedented progress due to the development of diffusion models. Beyond text and image, sound is a vital element within the sphere of human perception, offering vivid representations and naturally coinciding with corresponding scenes. Taking advantage of sound therefore presents a promising avenue for exploration within image generation research. However, the relationship between audio and image supervision remains significantly underdeveloped, and the scarcity of related, high-quality datasets brings further obstacles. In this paper, we propose a unified framework 'Align, Adapt, and Inject' (AAI) for sound-guided image generation, editing, and stylization. In particular, our method adapts input sound into a sound token, like an ordinary word, which can plug and play with existing powerful diffusion-based Text-to-Image (T2I) models. Specifically, we first train a multi-modal encoder to align audio representation with the pre-trained textual manifold and visual manifold, respectively. Then, we propose the audio adapter to adapt audio representation into an audio token enriched with specific semantics, which can be injected into a frozen T2I model flexibly. In this way, we are able to extract the dynamic information of varied sounds, while utilizing the formidable capability of existing T2I models to facilitate sound-guided image generation, editing, and stylization in a convenient and cost-effective manner. The experiment results confirm that our proposed AAI outperforms other text and sound-guided state-of-the-art methods. And our aligned multi-modal encoder is also competitive with other approaches in the audio-visual retrieval and audio-text retrieval tasks.