Multimodal Deep Generative Model for Semi-Supervised Learning under Class Imbalance

TL;DR

This paper introduces a multimodal deep generative model designed for semi-supervised learning on imbalanced datasets, leveraging separate modality encoders, Student's t-distributions, and a novel training objective to improve classification accuracy.

Contribution

It proposes a new multimodal generative framework with heavy-tailed priors and a specialized training objective to effectively handle class imbalance and partial supervision.

Findings

Outperforms baseline methods on benchmark datasets.

Achieves superior classification accuracy with imbalanced, partially labeled multimodal data.

Demonstrates robustness across real-world datasets.

Abstract

When modeling class-imbalanced data, it is crucial to address the imbalance, as models trained on such data tend to be biased towards the majority classes. This problem is amplified under partial supervision, where pseudo-labels for unlabeled data are predicted based on imbalanced labeled data, propagating the bias. While recent semi-supervised models address class imbalance, they typically assume single-modal input data. However, with the growing availability of multimodal data, it is essential to leverage complementary modalities. In this article, we propose a multimodal deep generative model for semi-supervised learning under class imbalance. Our approach uses separate encoders for each modality, sharing latent variables across modalities, and simplifies joint posterior computation with a product-of-experts method. To further address class imbalance, we replace typical Gaussian distributions with Student's t-distributions for the prior, encoder, and decoder, better capturing the heavy-tailed latent distributions in imbalanced data. We derive a new objective function for training the proposed model on both labeled and unlabeled data using $\gamma$-power divergence. Empirical results on benchmark and real-world datasets demonstrate that our model outperforms baseline methods in generalization, achieving superior classification performance for partially labeled multimodal data with imbalanced class distributions.