Multimodal Classification via Total Correlation Maximization

TL;DR

This paper introduces TCMax, a novel method for multimodal classification that maximizes total correlation between features and labels, effectively balancing modalities and capturing inter-modal interactions, leading to improved performance.

Contribution

The paper proposes TCMax, a new hyperparameter-free loss function based on total correlation maximization, with a theoretical foundation and practical effectiveness demonstrated through extensive experiments.

Findings

TCMax outperforms existing methods in multimodal classification.

The approach effectively balances modality contributions and captures inter-modal interactions.

The method is hyperparameter-free and theoretically grounded.

Abstract

Multimodal learning integrates data from diverse sensors to effectively harness information from different modalities. However, recent studies reveal that joint learning often overfits certain modalities while neglecting others, leading to performance inferior to that of unimodal learning. Although previous efforts have sought to balance modal contributions or combine joint and unimodal learning, thereby mitigating the degradation of weaker modalities with promising outcomes, few have examined the relationship between joint and unimodal learning from an information-theoretic perspective. In this paper, we theoretically analyze modality competition and propose a method for multimodal classification by maximizing the total correlation between multimodal features and labels. By maximizing this objective, our approach alleviates modality competition while capturing inter-modal interactions via feature alignment. Building on Mutual Information Neural Estimation (MINE), we introduce Total Correlation Neural Estimation (TCNE) to derive a lower bound for total correlation. Subsequently, we present TCMax, a hyperparameter-free loss function that maximizes total correlation through variational bound optimization. Extensive experiments demonstrate that TCMax outperforms state-of-the-art joint and unimodal learning approaches. Our code is available at https://github.com/hubaak/TCMax.