Speak the Art: A Direct Speech to Image Generation Framework

TL;DR

This paper introduces Speak the Art, a speech-to-image generation framework that uses a speech encoder supervised by a large image-text model and a VQ-Diffusion network, achieving superior results and stability over prior methods.

Contribution

The paper proposes a novel speech-to-image framework combining supervised speech encoding and diffusion models, improving stability, diversity, and multilingual capabilities.

Findings

Outperforms state-of-the-art models significantly

Uses diffusion instead of GANs for stable training

Demonstrates multilingual speech-to-image generation

Abstract

Direct speech-to-image generation has recently shown promising results. However, compared to text-to-image generation, there is still a large gap to enclose. Current approaches use two stages to tackle this task: speech encoding network and image generative adversarial network (GAN). The speech encoding networks in these approaches produce embeddings that do not capture sufficient linguistic information to semantically represent the input speech. GANs suffer from issues such as non-convergence, mode collapse, and diminished gradient, which result in unstable model parameters, limited sample diversity, and ineffective generator learning, respectively. To address these weaknesses, we introduce a framework called Speak the Art (STA) which consists of a speech encoding network and a VQ-Diffusion network conditioned on speech embeddings. To improve speech embeddings, the speech encoding network is supervised by a large pre-trained image-text model during training. Replacing GANs with diffusion leads to more stable training and the generation of diverse images. Additionally, we investigate the feasibility of extending our framework to be multilingual. As a proof of concept, we trained our framework with two languages: English and Arabic. Finally, we show that our results surpass state-of-the-art models by a large margin.