Unconditional Image-Text Pair Generation with Multimodal Cross Quantizer

TL;DR

This paper introduces MXQ-VAE, a novel multimodal cross-quantization VAE that enables unconditional generation of semantically consistent image-text pairs by learning a joint representation space.

Contribution

The paper proposes a new vector quantizer for joint image-text representations and demonstrates its effectiveness for unconditional multimodal pair generation.

Findings

Joint image-text representation space is effective for semantically consistent generation.

The method outperforms several baselines on synthetic and real-world datasets.

The approach enables unconditional generation of image-text pairs with semantic coherence.

Abstract

Although deep generative models have gained a lot of attention, most of the existing works are designed for unimodal generation. In this paper, we explore a new method for unconditional image-text pair generation. We design Multimodal Cross-Quantization VAE (MXQ-VAE), a novel vector quantizer for joint image-text representations, with which we discover that a joint image-text representation space is effective for semantically consistent image-text pair generation. To learn a multimodal semantic correlation in a quantized space, we combine VQ-VAE with a Transformer encoder and apply an input masking strategy. Specifically, MXQ-VAE accepts a masked image-text pair as input and learns a quantized joint representation space, so that the input can be converted to a unified code sequence, then we perform unconditional image-text pair generation with the code sequence. Extensive experiments show the correlation between the quantized joint space and the multimodal generation capability on synthetic and real-world datasets. In addition, we demonstrate the superiority of our approach in these two aspects over several baselines. The source code is publicly available at: https://github.com/ttumyche/MXQ-VAE.