Multimodal Generative Learning Utilizing Jensen-Shannon-Divergence

TL;DR

This paper introduces a new efficient multimodal generative learning method using Jensen-Shannon divergence, which improves joint distribution learning and dependency modeling across different data types.

Contribution

It proposes a novel mmJSD objective that directly approximates unimodal and joint posteriors, with theoretical proof of ELBO optimization and improved experimental performance.

Findings

Outperforms previous models in unsupervised generative tasks

Efficient training scheme for multimodal learning

Theoretically proven to optimize an ELBO

Abstract

Learning from different data types is a long-standing goal in machine learning research, as multiple information sources co-occur when describing natural phenomena. However, existing generative models that approximate a multimodal ELBO rely on difficult or inefficient training schemes to learn a joint distribution and the dependencies between modalities. In this work, we propose a novel, efficient objective function that utilizes the Jensen-Shannon divergence for multiple distributions. It simultaneously approximates the unimodal and joint multimodal posteriors directly via a dynamic prior. In addition, we theoretically prove that the new multimodal JS-divergence (mmJSD) objective optimizes an ELBO. In extensive experiments, we demonstrate the advantage of the proposed mmJSD model compared to previous work in unsupervised, generative learning tasks.