Generalizing Multimodal Variational Methods to Sets

TL;DR

This paper introduces SMVAE, a novel variational approach that models joint multimodal posteriors directly, enabling order-agnostic cross-modal generation and improved performance over existing methods.

Contribution

The paper proposes SMVAE, a set-based variational method that directly models joint multimodal posteriors, overcoming limitations of previous uni-modality approximations.

Findings

Effective order-agnostic cross-modal generation demonstrated

Outperforms state-of-the-art multimodal methods on various datasets

Handles missing modality problem successfully

Abstract

Making sense of multiple modalities can yield a more comprehensive description of real-world phenomena. However, learning the co-representation of diverse modalities is still a long-standing endeavor in emerging machine learning applications and research. Previous generative approaches for multimodal input approximate a joint-modality posterior by uni-modality posteriors as product-of-experts (PoE) or mixture-of-experts (MoE). We argue that these approximations lead to a defective bound for the optimization process and loss of semantic connection among modalities. This paper presents a novel variational method on sets called the Set Multimodal VAE (SMVAE) for learning a multimodal latent space while handling the missing modality problem. By modeling the joint-modality posterior distribution directly, the proposed SMVAE learns to exchange information between multiple modalities and compensate for the drawbacks caused by factorization. In public datasets of various domains, the experimental results demonstrate that the proposed method is applicable to order-agnostic cross-modal generation while achieving outstanding performance compared to the state-of-the-art multimodal methods. The source code for our method is available online https://anonymous.4open.science/r/SMVAE-9B3C/.