Multimodal Generative Models for Compositional Representation Learning

TL;DR

This paper introduces a family of multimodal deep generative models that effectively combine image and text data, improving representation learning and downstream task performance through novel variational objectives and model combinations.

Contribution

It presents a new variational bound-based framework for multimodal generative models, generalizes to various deep generative types, and demonstrates improved performance and interpretability across multiple datasets.

Findings

Multimodal VAEs outperform previous models with and without weak supervision.

Combining GANs with VAEs enhances image and text generation quality.

Language influences image representations, making them more abstract and compositional.

Abstract

As deep neural networks become more adept at traditional tasks, many of the most exciting new challenges concern multimodality---observations that combine diverse types, such as image and text. In this paper, we introduce a family of multimodal deep generative models derived from variational bounds on the evidence (data marginal likelihood). As part of our derivation we find that many previous multimodal variational autoencoders used objectives that do not correctly bound the joint marginal likelihood across modalities. We further generalize our objective to work with several types of deep generative model (VAE, GAN, and flow-based), and allow use of different model types for different modalities. We benchmark our models across many image, label, and text datasets, and find that our multimodal VAEs excel with and without weak supervision. Additional improvements come from use of GAN image models with VAE language models. Finally, we investigate the effect of language on learned image representations through a variety of downstream tasks, such as compositionally, bounding box prediction, and visual relation prediction. We find evidence that these image representations are more abstract and compositional than equivalent representations learned from only visual data.