Joint Multimodal Learning with Deep Generative Models

TL;DR

This paper introduces JMVAE, a deep generative model that learns a joint representation for multiple modalities, enabling bi-directional generation and reconstruction of modalities such as images and text.

Contribution

The paper proposes a novel joint multimodal variational autoencoder (JMVAE) and an enhanced version JMVAE-kl, which effectively model multi-modal data and generate missing modalities.

Findings

JMVAE captures high-level joint representations of multiple modalities.

JMVAE outperforms conventional VAEs in generating and reconstructing modalities.

JMVAE can generate multiple modalities in both directions.

Abstract

We investigate deep generative models that can exchange multiple modalities bi-directionally, e.g., generating images from corresponding texts and vice versa. Recently, some studies handle multiple modalities on deep generative models, such as variational autoencoders (VAEs). However, these models typically assume that modalities are forced to have a conditioned relation, i.e., we can only generate modalities in one direction. To achieve our objective, we should extract a joint representation that captures high-level concepts among all modalities and through which we can exchange them bi-directionally. As described herein, we propose a joint multimodal variational autoencoder (JMVAE), in which all modalities are independently conditioned on joint representation. In other words, it models a joint distribution of modalities. Furthermore, to be able to generate missing modalities from the remaining modalities properly, we develop an additional method, JMVAE-kl, that is trained by reducing the divergence between JMVAE's encoder and prepared networks of respective modalities. Our experiments show that our proposed method can obtain appropriate joint representation from multiple modalities and that it can generate and reconstruct them more properly than conventional VAEs. We further demonstrate that JMVAE can generate multiple modalities bi-directionally.