CoVAE: correlated multimodal generative modeling

TL;DR

CoVAE introduces a novel generative model that captures correlations between modalities in multimodal data, improving reconstruction accuracy and uncertainty quantification compared to existing methods.

Contribution

This work presents CoVAE, a new architecture that preserves joint statistical structure in multimodal data, addressing limitations of previous latent space fusion strategies.

Findings

Accurate cross-modal reconstruction demonstrated.

Effective uncertainty quantification achieved.

Works on both real and synthetic datasets.

Abstract

Multimodal Variational Autoencoders have emerged as a popular tool to extract effective representations from rich multimodal data. However, such models rely on fusion strategies in latent space that destroy the joint statistical structure of the multimodal data, with profound implications for generation and uncertainty quantification. In this work, we introduce Correlated Variational Autoencoders (CoVAE), a new generative architecture that captures the correlations between modalities. We test CoVAE on a number of real and synthetic data sets demonstrating both accurate cross-modal reconstruction and effective quantification of the associated uncertainties.